Rusty CPython Iterator

Python のイテレーターと Rust の借用


Who am I?


  • Yuya Nishihara

    twitter: @yujauja

  • Embedded software engineer
  • image

    Mercurial committer

    • DVCS implemented in Python
      • with C
      • and Rust

Rust-Python Bindings

Two major crates:


PyO3 "minimum required Rust version is 1.37.0-nightly"

rust-cpython Basics

which (basically) generates:

  • All data backed by CPython memory (i.e. reference counted.)
  • Any data field must be Send + 'static,
  • py: Python<'a> (!Sync nor !Send) marker guarantees GIL acquired,
  • so that IntSet can be Send and Sync.


We need iterator!
Both Rust and Python have.

Python iterator

over Rust collection:


The Problem

<'a> cannot live in the Python world.

Someone says

but <'static> can!!

unsafe { &*ptr }

The idea

Cheated lifetime so the iterator can (out)live in the Python world.

How to make it safe?

Things to consider:

  • Real object lifetime
  • Multi-threading stuff
  • Mutation of underlying object
  • Safe interface on top of these three

Object lifetime

Incref the owner PyObject and keep it.

Multi-threading stuff

Easy. Just require GIL for any operation.

Mutation #1

Invalidate references when underlying object mutated.

Mutation #2

Prevent mutation while iterator is borrowed,
just like RefCell.

Safe interface

Safe interface is not safe

because the lifetime is cheated.

Many ways to obtain the underlying &'static:

Lifetime is hard

&'static is not a silver bullet.

Source code

Further Reading