Goals for Python 3.5

Python 3.4 is in its first beta and the release is just around the corner.  Hurray!  That means we are in feature freeze.  While I am finishing up odds and ends for the PEP 451 (module specs) implementation, I'm also starting to think seriously about what I hope to accomplish for the next Python version, 3.5.

The following are lists of things I'd like to make happen for Python 3.5.  Most of them are things I wanted to do for 3.4 but ran out of time (go figure).  I'll be adding to this list as I run across things I missed.  I'm sure I've missed a thing or three that I promised myself or someone else that I'd work on.

Inspired in part by Nick Coghlan, I've categorized this 3.5 wishlist in descending order of likelihood.

[WARNING: Realistically, a bunch of this will not happen.]

Must-haves

These things are my main priorities.  Barring rejection in python-dev, I will work to push these through.

1. a C implementation of OrderedDict (basically done)
2. OrderedDict as the default class definition namespace [guido]
3. a __definition_order__ attribute on classes that preserves the order of keys (or at least the keys) [me]
4. a __kwargs_order__ attribute in function locals that preserves the order of keys from **kwargs in calls (in lieu of making kwargs an OrderedDict) [link]
5. a new type that abstracts the full import system (similar to PEP 406)
    - incl. exposing suffix->loader mapping
    - context manager support
6. add statement local namespaces [PEP 3150][PEP 403][PEP 340][PEP 359][Nick]

#1-4 are all related, though 4 probably won't involve OrderedDict when all is said and done.  They shouldn't require all that much work to take across the finish line.

Helping Out

These are things that I expect others will be targeting for 3.5 and with which I plan on helping.

* refactor pythonrun.c [PEP 432]
* refactor the C extension module API to take advantage of the PEP 451 loader APIs [import-sig link][issue16421][issue13429][issue16392?]
    - incl. implement exec_module() for BuiltinImporter and ExtensionFileLoader
* a successor to PEP 395
* "speed up" interpreter start-up, import [issue16101][link][link]
* [subinterpreters] (incl. using them to circumvent GIL? [link], interpreter-per-thread)
* a pure python zip importer [link]
* a lazy importer [link]
* missing *-unpacking generalizations [PEP 448]
* add --path0/--no-path0 cmdline options (also issue15716)
* add __locallookup__ method to metaclass [PEP 447]?

* address the hacky uses of function parameter defaults [link] (Jan K.)

Smaller Tasks

These are things that I hope to get done for 3.5, but won't require a lot of work.

* a new frame attribute, f_func, which is a weakref to the function object that was actually called (basically done)
* give the sys module a custom type and add descriptors [link]
* add inspect.Signature.from_callable()
* change setup.py (the main one) to not rely on imp.load_dynamic() [imp => _imp?] (may be done already)
* update pickle to use __spec__ (in the __main__ case)

Would be Nice

Stuff that I'd like to see in 3.5 but that I likely won't have time to work on.

* interpreter startup hook (via cmdline option) [link] (for coverage.py, etc.)
* allow classes to be referenced in the class body
* extension/plugin framework for unittest a la nose [link]

May Be a Good Idea...

Other things that may or may not be worth doing.

* expose _SpecMethods (or something like it) publicly
* add importlib.util.load_from_spec()?
* .ref files [link]

* find a non-hacky replacement for a module replacing itself in sys.modules [link]
* re-organize the sys module with sub-namespaces along the lines of the config categories listed in PEP 432 (similar to but more extensive than PEP 3139)

* parser hooks
* tokenizer hooks
* traceback hooks
* API for excluding functions/methods from tracebacks like importlib and unittest do [issue1705520?]

* spec.data (a wrapper for the data-related API of loaders) [issue14982?]

* add a classonlymethod decorator (not necessarily as a builtin) [link]
* support positional-only parameters (via decorator?) [link][PEP 457]
* add a listproxy type to parallel dictproxy?
* improve re-import of builtin modules, extension modules (see above in Helping Out) [issue14715?]
* generic context API (incl. Context, ThreadLocalContext), a la decimal (email policy?) (asyncio concerns [link])
* post-import hooks (a la PEP 369)
* abc.implements(obj, *<ABC>) that ensures the object has all the appropriate attributes
* ABCMeta.implements(decorated) that registers the decorated class on an ABC and wraps the decorated class's __init__() with an abc.implements() call after it runs [issue12128]
* some mechanism to inherit docstrings [issue15582][issue15731]
* expose PyArg_ParseTuple, etc. in Python
* function to resolve __qualname__ (or provide func.__origin__)

More Radical Ideas

* always import from <module.py> and set __path__ if there is a directory by the same name (obviates __init__.py) [link] (related to PEP 402 discussions) -- at least provide an import hook
* provide a safe mode for ImportSystem (see #5 above) that prevents leaking import side effects outside a context

* replace "if __name__ == '__main__'" [link][PEP 299][PEP 3122] (__spec__.is_main + builtin is_main?)

* a reverse name binding protocol (the object being bound/unbound to the name is notified) [link]

* expose key metaclass operations via special class methods (a la PEP 422)
* attribute docstrings [link]
* a hook to process a docstring after its corresponding object is created (but before decorators are applied)

* revisit PEP 224, PEP 329, PEP 351, PEP 363, PEP 377 (Kristjan)

Random Off-the-cuff Ideas

* a decorator for type hints (optional static typing)
* a decorator that supports all sorts of *args/**kwargs handling
* define a standard API for object (modules included) to "install" themselves; provide installer helpers (a la install_importhook() or an Installer base class + ImportHookInstaller base subclass)
* generic Registry class (register() method, etc.)
* spec.open() for locatable specs
* expose an easy way to make some module-global read-only (make use of loader.create_module()?) [issue15031]
* break or return in global module code to stop early (instead of delegating skipped code to submodule)
* ...

This list is as endless as the sea! :-)

My PyCon 2012 Talk Proposals

For the first time I have actually submitted a talk proposal for PyCon.  Well, actually I submitted 8.  It'll help my odds!  <wink>  I figured I would post the proposals here, so people would know what's been eating a lot of my time for the last little while.

I'm showing the proposals in my order of preference.  I really hope that one of my top three gets in, but any of them would be awesome.  Regardless of how it goes with the proposals, I'll see about presenting each of these at the Utah Python Users Group.

If you have any thoughts or questions, simply leave a comment. :)

Here's the list:

* Getting the Most Out of Python Imports

* The Future of Python's Imports
* Explicit Handlers to Python Language Features
* Interfaces and Python
* AST Transformations using PEP 302
* Code Archaeology and Repository Spelunking
* Running Python in Your Browser with Chrome's NativeClient
* Python on Android

Getting the Most Out of Python Imports

To really take advantage of Python you must understand how imports work and how to use them effectively. In this talk we'll discuss both of these. After a short introduction to imports, we'll dive right and show how you can put PEP 302 import hooks to work for you.

Type: Talk

Python's import statement has been a powerful feature since the first release, and only gotten better with age. Understanding how imports work under the hood will let you take advantage of that power.

The key to customizing Python's imports is the importers introduced by PEP 302. That's a tool that you want in your belt!

Talk Outline

1. Python's imports (5 min)
   • the evolution of the import machinery
   • the gears in the machine
   • motivation for PEP 302
2. Finder Objects (5 min)
   • what they're for
   • how to write one
   • what to do with it
3. Loader Objects (5 min)
   • what they're for
   • how to write one
   • what to do with it
4. sys.meta_path vs. sys.path_hooks (5 min)
   • when and how to use them
   • example of putting it all together
5. Real-life Examples of Import Hooks (7 min)
   • Python/import.c
   • importlib module
   • PyPy
   • PyFilesystem
6. Other Import Customizations (3 min)
   • builtins.__import__
   • .pth files
   • PEP 402
   • import engine

The Future of Python's Imports

The history of Python's import machinery is one of evolving power since its humble beginnings in the original Python release. That trend is continuing with efforts to make Python's imports even more useful. Come hear about the current and upcoming project, and learn why it matters to you.

Type: Talk

Here's an outline of the talk:

1. Imports in Python (5 min)
   • the evolution of the import machinery
   • the gears in the machine
2. importlib (7 min)
   • its introduction in the stdlib
   • as the default builtins.__import__
   • status
3. PEP 382/402 (5 min)
   • why "namespace" packages matter
   • how it would work
   • status
4. The Import Engine (5 min)
   • consolidating the import state
   • how it could help you
   • status
5. Other PEPs (3 min)
   • PEP 369 -- Post import hooks
   • PEP 395 -- Module Aliasing
6. And Beyond... (5 min)
   • Armin's Rant
   • exocet, mercurial, and PEAK

Much more information on Python imports may be found at my Python Imports page.

Explicit Handlers to Python Language Features

Come learn all about how Python's language features are handled and how some of that behavior can be customized. We'll go from the grammar to AST and on to the functions that handle the behavior.

Type: Talk

Like all programming languages, Python can be described by the language features it has and how it handles those features. One of Python's strengths is how some of these handlers, like __import__ or __len__, can be explicitly customized to extend the language.

In this talk we'll look at Python's language features and how it handles them. We'll trace the path from syntax through AST and beyond. We'll also look at the handlers that you can override and what implicit handlers are candidates to be exposed. This talk will be partly CPython-specific, but the analytic process applies equally to alternate implementations.

Talk Outline:

1. Language Features and Handlers (3 min)
   • what is a language feature?
   • what is a handler?
   • a comparison across languages of a feature subset
2. Python Language Features (7 min)
   • Python's language feature catalog
   • a handler for each feature
   • examples of implicit handlers
   • examples of explicit handlers (special methods)
3. From Grammar to AST (6 min)
   • mapping features to grammar
   • mapping grammar to AST
   • some concrete examples
   • examining the transformation
4. From AST to Handler (9 min)
   • via opcodes in CPython
   • mapping AST to opcodes (compiler)
   • mapping opcodes to handlers
   • implicit handlers as an implementation detail
   • some concrete examples
   • examining the transformation
5. Current Implicit Handlers (5 min)
   • a look at the handlers that are not customizable
   • which ones could be turned into explicit handlers
   • the impact of doing so

More information on Language Feature Handlers see my Python Language Feature Handlers page.

Interfaces and Python

In 2.6, Python introduced the Abstract Base Classes. Before that we had "protocols" (and we still do). In this talk we'll look at the how the general concept of interfaces fits into today's Python. We'll also look at some of the alternate proposals of the past, some of the controversies around ABCs, and the direction interfaces might go in the future.

Type: Talk

Talk Outline:

1. What are Interfaces? (3 min)
   • modeling strict abstraction
   • precedents in other languages
2. Interfaces in Python (6 min)
   • duck-typing
   • Python "protocols"
   • past proposals (PEP 245)
   • how Python "interfaces" are different
3. Newer Interface Support (11 min)
   • annotations
   • Abstract Base Classes
   • why run-time validation?
   • ABC vs. duck-typing
4. Third-party Libraries (5 min)
   • Peak's PyProtocols
   • zope.interface
   • Twisted
5. What Next? (3 min)
   • strict interfaces
   • compile-time validation
   • an example interface library

For more comprehensive coverage of interfaces in Python, check out this reference.

AST Transformations using PEP 302

Python's stdlib offers the ast module, which exposes the AST portion of the compiler. It's a powerful tool for manipulating code before compilation. Combine this with a PEP 302 import hook and you are ready to do some pretty neat stuff. We'll use domain-specific languages to demonstrate the power of this technique.

Type: Talk

AST + PEP 302 == awesome

First of all, if you want an introduction to the stdlib ast module or PEP 302 importers, this talk has a lot to offer. If you are interested in domain-specific languages, we'll be talking about those too.

The magic of combining the ast module with import hooks is in the that ability to transform a seemingly invalid module into a valid one. The end result is compiled python code, as though you had written the module in legal Python in the first place.

Talk Outline:

1. the stdlib ast module (5 min)
   • briefly cover CPython's compiler
   • look at the interface of the AST module
   • a quick example of using the AST module to modify partially compiled code
2. PEP 302 importers (5 min)
   • customizing imports and the problem with builtins.import
   • what goes into an importer (finders and loaders)
   • the default importers
   • sys.meta_path vs. sys.path_hooks
   • two examples of custom importers
3. an example of a domain-specific language (5 min)
   • what is a DSL?
   • a simple DSL for SQL
4. putting it all together (15 min)
   • an AST transformer for the DSL
   • an importer that intelligently applies the AST transformer
   • actually using it with a real database

For more information, go to my pages on Code Transformations in Python, Python Imports, and Domain-Specific Languages in Python.

Code Archaeology and Repository Spelunking

Come learn about the tools I used and experience I had while peeling back the layers of CPython's full repository. We'll also talk about how these relate to "code archaeology" in general.

Type: Talk

The power of version control lies partly in the history it offers. However, for a large project digging down into the repository to gather historical artifacts is no trivial matter.

While working on a project related to Python's history, I needed to look at commit history and to search through commits/patches. Thankfully CPython's repository goes back almost to the beginning (1990). In early 2011 the CPython repository moved to Mercurial, which factors in to the tools I was able to use in my spelunking.

In this talk we'll look at the tools and methods I used in my "archaeology", with a focus on Mercurial and CPython. However, the material should be applicable to most VCSs and most projects.

Talk Outline:

1. What is Code Archaeology? (3 min)
   • motivation (understanding what brought us here)
   • why does it matter?
2. The Tools (2 min)
   • VCS
   • mail archives
3. Investigating CPython (2 min)
   • CPython resources
   • mercurial repos
   • mail archives
4. The Evolution of a Code Base (3 min)
   • how projects evolve
   • examples from CPython
5. The Players (3 min)
   • personalities and community play a part
   • examples from CPython
6. Preparing to Dive In (3 min)
   • determine what you care about
   • know what to look for (get familiar with the topic)
   • examples from CPython
7. Extracting Timelines (7 min)
   • take a focused approach
   • search permutations on the VCS
   • supplement with other resources
   • examples from CPython
8. The Story Behind the Commits (4 min)
   • lingering questions (like "why?!?")
   • searching in the mail archives
   • timeline offers a target
   • examples from CPython
9. What Might Have Been (3 min)
   • alternate outcomes mostly fade into obscurity
   • lessons learned from them
   • examples from CPython
10. What I learned about the CPython Core Developers (2 min)

For more information go to my Code Archaeology and Repository Spelunking page.

Running Python in Your Browser with Chrome's NativeClient

Chrome's Native Client has gotten a lot of press in the last year. It's a tool for compiling C/C++ to native code and running it sandboxed in your browser. This is a talk about porting Python to run in the Native Client, and why you'd like that.

Type: Talk

Mark Seaborn from the Chromium project has done a lot of work on getting CPython to run inside Chrome's Native Client. In this talk we'll look at the work he's done, what's left to do, and how you can help.

We'll also talk about why Python in the NaCl sandbox matters and we'll wrap up by discussing the idea of PyPy on Native Client.

Talk Outline:

1. What is Native Client? (5 min)
   • virtual machine for C/C++
   • released on Chrome in 2011
   • continued work to port libraries
   • uses Python 2.x for a number of tools
2. Porting CPython to Native Client (10 min)
   • initial work by Mark Seaborn (Jun. 2009)
   • trouble with dynamic linking and build tools (Dec. 2010)
   • upcoming dynamic linking support and Python bindings in NaCl
   • current roadblocks and options
3. Possibilities with PyPy (15 min)
   • using PyPy's RPython toolchain to port Python to NaCl
   • is a NaCl backend for PyPy pointless?
   • porting the toolchain to NaCl
   • examples

Python on Android

Come learn about the present and future of writing Android apps in Python. We'll cover SL4A and efforts to port both Jython and PyPy.

Type: Talk

The Android mobile operating system is a great target for developers. However, when you write apps for the dalvik virtual machine, you have to write in Java. Personally, I would rather not. Instead, wouldn't it be nice to write Android apps in Python? This has certainly crossed every Python programmer's mind who has even thought about Android. So what are the options?

First of all, in 2008 there was a project called jythonroid that tried to port Jython to Android. We'll talk about why it didn't pan out.

Secondly, in 2010 Google released the "Scripting Layer for Android" (SL4A) project with Python support; and in 2011 they spun off the Python portion into its own project. It's neat to be able to write Python on my phone. We'll talk about why this currently isn't a good solution for writing Android Apps; and what could make it work better (i.e. a tool for building wrappers around SL4A scripts).

Finally, two of the Python implementations have already been involved in discussions on porting Python to Android: Jython and PyPy. We'll talk about what happened with past (official) porting efforts for Jython and where efforts are headed for both projects.

The idea of writing Android apps in Python is both appealing and elusive. It invites your imagination. So, let's tap into that! To wrap up the talk we'll look at what it would be like to write for Android in Python and what you can do to help make that a reality.

Talk Outline:

1. Programming for Android (3 min)
   • Java
   • Android API
   • an example
   • Dalvik is not JVM
2. Python on Android: SL4A (6 min)
   • summary
   • examples
   • difference from native Android apps
   • why SL4A Isn't Good Enough
   • making it a little better
3. Current Efforts in Jython (13 min)
   • jythonroid (2009)
   • jython-for-android (2011)
   • early optimism (2008/2009)
   • hints of Android support for 2.5.1 (2009)
   • The challenge of dynamic code generation on Android
   • PBC (Python bytcode) and Java PBC VM
   • performance implications
   • examples (theoretical)
   • future availability
4. Current Efforts in PyPy (8 min)
   • acknowledged early (2009)
   • JIT backend for ARM (2011)
   • examples (theoretical)
   • future availability