# Allison Kaptur In early October I gave a keynote at Python Brasil in Belo Horizonte. Here is an aspirational and lightly edited transcript of the talk. There is ... This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Lessons from Investigating Software Bugs, Including Dropbox's File-Sync Architecture DevFeed: [Lessons from Investigating Software Bugs, Including Dropbox's File-Sync Architecture]() Original publisher: [Read original article]() Published: 2017-11-13T02:23:00Z Content type: opinion Language: en Sources: [Allison Kaptur]() Topics: [bug](), [dropbox](), [Filesystems](), [Back end](), [Cloud](), [client]() Tags: [architecture](), [backend](), [bugs](), [client](), [cloud](), [dropbox](), [file]() ### AI overview A keynote transcript argues that investigating software bugs can be entertaining and instructive. It introduces a Dropbox bug and describes a simplified architecture in which a desktop client detects filesystem changes, hashes files into 4MB blocks, and stores those blocks in a backend key-value store. ### Source excerpt In early October I gave a keynote at Python Brasil in Belo Horizonte. Here is an aspirational and lightly edited transcript of the talk. There is also a video available here. I love bugs I'm currently a senior engineer at Pilot.com, working on automating bookkeeping for startups. Before that, I worked for Dropbox on the desktop client team, and I'll have a few stories about my work there. Earlier, I was a facilitator at the Recurse Center, a writers retreat for programmers in NYC. I studied astrophysics in college and worked in finance for a few years before becoming an engineer. But none of that is really important to remember - the only thing you need to know about me is that I love bugs. I love bugs because they're entertaining. They're dramatic. The investigation of a great bug can be full of twists and turns. A great bug is like a good joke or a riddle - you're expecting one outcome, but the result veers off in another direction. Over the course of this talk I'm going to tell you about some bugs that I have loved, explain why I love bugs so much, and then convince you that you should love bugs too. Bug #1 Ok, straight into bug #1. This is a bug that I encountered while working at Dropbox. As you may know, Dropbox is a utility that syncs your files from one computer to the cloud and to your other computers. 1 2 3 4 5 6 7 8 9 10 11 12 +--------------+ +---------------+ | | | | | METASERVER | | BLOCKSERVER | | | | | +-+--+---------+ +---------+-----+ ^ | ^ | | | | | +----------+ | | +---> | | | | | CLIENT +--------+ +--------+ | +----------+ Here's a vastly simplified diagram of Dropbox's architecture. The desktop client runs on your local computer listening for changes in the file system. When it notices a changed file, it reads the file, then hashes the contents in 4MB blocks. These blocks are stored in the backend in a giant key-value store that we call blockserver. The key is the digest of the hashed contents, and the values are the contents themselves. Of cou ## Effective Learning Strategies for Programmers DevFeed: [Effective Learning Strategies for Programmers]() Original publisher: [Read original article]() Published: 2015-10-10T22:15:00Z Content type: article Language: en Sources: [Allison Kaptur]() Topics: [Learning](), [Programming]() Tags: [effective](), [learning](), [programming](), [research](), [skills](), [strategies]() ### AI overview An article about effective learning for programmers, presented as an edited transcript of a Kiwi PyCon keynote. It introduces mindset and learning strategies, drawing on experiences at the Recurse Center and research on effective learning. ### Source excerpt In early September I gave a keynote at Kiwi PyCon in New Zealand on effective learning for programmers. There were two pieces to the talk: one about mindset, and one about particular strategies we can use. The text below is an aspirational and lightly edited transcript of the mindset piece of that talk. There's also a video available if you'd like to see the strategies piece. Recurse Center Before I joined Dropbox last year, I spent two years working at a company in NYC called the Recurse Center. The Recurse Center is like a writers' retreat for programmers. Participants spend 3 months working on whatever is most interesting to them. So someone who'd been writing Java for ten years might come to RC to learn a new language like Clojure, or someone who just graduated with a CS degree might come work on their web development skills, or someone who'd been learning programming in their spare time might come to turbo-charge their learning. There's almost no structure to the program - no deadlines, no assignments, no teaching. It's an experiment in unstructured learning for adults. My role as a facilitator was to help people make the most of that disorienting amount of freedom that they had at RC. People who come out of traditional educational experiences or traditional jobs very often don't know what to do with that. So I'd help them with goal-setting and help them make the most of the experience. One of the things we thought a lot about was how to have the most effective learning experience possible for programmers. Today I'll talk about some of the research into how to be an effective learner, and how we can apply that research to our daily lives as programmers and engineers. What to get out of this post Take a minute and consider what you'd like to get out of this post. You might want to learn something new about how to be as efficient and effective in your job as possible. You might want to hear about how you can be a better teacher or mentor to junior engineers. Or y ## PS1 for Python3 DevFeed: [PS1 for Python3]() Original publisher: [Read original article]() Published: 2014-10-23T21:38:00Z Content type: tutorial Language: en Sources: [Allison Kaptur]() Topics: [Python](), [version](), [syntax](), [function](), [export](), [file](), [import]() Tags: [export](), [file](), [function](), [import](), [python](), [syntax](), [version]() ### AI overview A Python developer explains how to customize the interactive prompt with PYTHONSTARTUP so it clearly indicates whether a session is running Python 2 or Python 3. The article also shows how to use print_function from __future__ to adopt Python 3-style print syntax in Python 2. ### Source excerpt I spend a lot of time flipping back and forth between Python 2.x and 3.x: I use different versions for different projects, talk to people about different versions, explore differences between the two, and paste the output of REPL sessions into chat windows. I also like to keep long-running REPL sessions. These two activities in combination became quite confusing, and I'd often forget which version I was using. 1 2 3 4 5 6 >>> print some_var File "", line 1 print some_var ^ SyntaxError: invalid syntax >>> # *swears* After the hundredth time I made this mistake, I decided to modify my prompt to make it always obvious which version was which, even in long-running REPL sessions. You can do this by creating a file to be run when Python starts up. Add this line to your .bashrc: 1 export PYTHONSTARTUP=~/mystartupscript.py Then in mystartupscript.py: mystartupscript.py 1 2 3 4 import sys if sys.version_info.major == 3: sys.ps1 = "PY3 >>> " sys.ps2 = "PY3 ... " This makes it obvious when you're about to slip up: 1 2 3 PY3 >>> for value in giant_collection: PY3 ... print(value) PY3 ... I've also add this line to mystartupscript.py to bite the bullet and start using print as a function everywhere: mystartupscript.py 1 from __future__ import print_function This has no effect in Python3.x, but will move 2.x to the new syntax. ## I'm joining Dropbox DevFeed: [I'm joining Dropbox]() Original publisher: [Read original article]() Published: 2014-10-10T18:49:00Z Content type: opinion Language: en Sources: [Allison Kaptur]() Topics: [dropbox](), [Python]() Tags: [community](), [dropbox](), [inclusive](), [leaving](), [python](), [work]() ### AI overview The author announces that they are leaving Hacker School to join Dropbox in San Francisco on Jessica McKellar's team. They reflect on their time facilitating at Hacker School, how it helped them become a programmer, and their continued connection to its community. ### Source excerpt Big news for me: I'm leaving Hacker School and going to work for Dropbox in San Francisco, joining Jessica McKellar's team. I met Jessica when she was part of the first round of residents at Hacker School in fall 2012, and I've had tremendous respect for her work and leadership ever since. Dropbox has an impressive crop of Pythonistas (including Guido van Rossum, of course), and I couldn't be more excited to join. I'll be moving to San Francisco at the end of October. If you have recommendations for people to meet, places to go, or things to do, let me know! This means I'm leaving Hacker School, after more than two years facilitating. My last day will be October 24th. I love Hacker School, and I know I'm going to miss it. Hacker School is entirely responsible for the fact that I'm a programmer at all. I was working in a finance job and contemplating new careers when my brother saw this post about Hacker School's experiment with Etsy to get more qualified women into the summer 2012 batch. I read the post and the thoughtful, welcoming FAQ, then went home and picked up a Python book. Two months later, I started Hacker School. Hacker School is about becoming a better programmer, and there's no doubt that it's worked for me. For two years, I've had total freedom to chase down whatever weird thing catches my eye; I've worked with creative, hilarious, brilliant Hacker Schoolers and residents on a dizzying variety of projects; and I've been delighted to help build a more inclusive environment at Hacker School, although there's always more work to be done. (If you're a curious, sharp, and self-directed programmer, I can't recommend Hacker School enough.) I'm thankful that leaving my job at Hacker School doesn't mean leaving the Hacker School community. I'm trading in my faculty status and becoming one of hundreds of alumni around the world. I'll still be on Zulip, Community, and everywhere else Hacker Schoolers can be found, and I'll still have my cape. I may be leaving, but ## Debugging with pstree DevFeed: [Debugging with pstree]() Original publisher: [Read original article]() Published: 2014-09-22T00:46:00Z Content type: tutorial Language: en Sources: [Allison Kaptur]() Topics: [debugging](), [Bash](), [Ruby](), [Databases](), [Command-line interface]() Tags: [bash](), [command](), [database](), [debugging](), [ruby](), [terminal]() ### AI overview A debugging account about a local PostgreSQL database restore failure after an outdated psql version and a PATH change. The article investigates why Ruby rake tasks could not find pg_restore and identifies that they were running under sh rather than bash. ### Source excerpt I hit a very fun bug yesterday while trying to run a script that sends emails to certain subsets of Hacker Schoolers. When I tried to test the script locally, I discovered that one of the tables of the database, Batch, was missing from my local version. After briefly panicking and making sure that the actual site was still up, I could dig in. It turns out that my local version of psql was way out of date, and as of a few days ago we'd started using a data type that wasn't present in my old version. Because of that, creating that particular table failed when I pulled from the production database the night before. The failure was logged, but the output is so verbose that I didn't notice the problem. Both the diagnosis and the fix here were easy - I went back and read the logs, googled the data type that was raising an error, and then upgraded Postgres.app and psql. That's when the real trouble started. The new version of Postgres.app was placed in a new spot on the $PATH, as you'd expect, and the upgrade prompted me to change my .bashrc, which I did. But the rake tasks we use to manage local copies of the database errored out with this message: 1 2 $ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U `whoami` -d hackerschool latest.dump sh: pg_restore: command not found This was pretty clearly a $PATH problem. I tried the usual things first, like sourcing my .bashrc in the terminal I was using, closing the terminal and opening a new one, etc. None of that worked. One thing that jumped out to me was the sh in the error message. That was an indicator that rake wasn't using bash as a shell - it was using sh - which means my .bashrc wasn't setting the environment. Reading the rake task showed that it was a thin wrapper around lots of system calls via Ruby's system("cmd here"). I added the line system("echo $PATH") and verified that the new location of pg_restore wasn't in it. At this point I found I had lots of questions about the execution context of the ra ## Getting Started with Python Internals DevFeed: [Getting Started with Python Internals]() Original publisher: [Read original article]() Published: 2014-08-03T16:17:00Z Content type: tutorial Language: en Sources: [Allison Kaptur]() Topics: [Python](), [Code](), [Programming](), [Development]() Tags: [code](), [getting-started](), [internals](), [programming](), [python]() ### AI overview A tutorial for experienced Python programmers who want a better mental model of Python internals. It explains that understanding internals is not required for good Python development, but can sometimes improve programming, support contributions to Python, and provide an interesting way to explore the language. The article recommends setting up a local Python source tree and examining code through hands-on strategies. ### Source excerpt I talk to a lot of people at Hacker School and elsewhere who have been programming Python for some time and want to get a better mental model of what's happening under the hood. The words "really" or "why" often features in these questions - "What's really happening when I write a list comprehension?" "Why are function calls considered expensive?" If you've seen any of the rest of this blog, you know I love digging around in Python internals, and I'm always happy to share that with others. Why do this? First off, I reject the idea that you have to understand the internals of Python to be a good Python developer. Many of the things you'll learn about Python won't help you write better Python. The "under the hood" construction is specious, too - why stop at Python internals? Do you also need to know C perfectly, and the C compiler, and the assembly, and ... That said, I think you should dig around in Python - it sometimes will help you write better Python, you'll be more prepared to contribute to Python if you want to, and most importantly, it's often really interesting and fun. Setup Follow the instructions in the Python dev guide under "Version Control Setup" and "Getting the Source Code". You now have a Python that you can play with. Strategies 1. Naturalism Peter Seibel has a great blog post about reading code. He thinks that "reading" isn't how most people interact with code - instead, they dissect it. From the post: But then it hit me. Code is not literature and we are not readers. Rather, interesting pieces of code are specimens and we are naturalists. So instead of trying to pick out a piece of code and reading it and then discussing it like a bunch of Comp Lit. grad students, I think a better model is for one of us to play the role of a 19th century naturalist returning from a trip to some exotic island to present to the local scientific society a discussion of the crazy beetles they found: "Look at the antenna on this monster! They look incredibly ungainly but ## The CPython Peephole Optimizer and You DevFeed: [The CPython Peephole Optimizer and You]() Original publisher: [Read original article]() Published: 2014-08-02T18:25:00Z Content type: tutorial Language: en Sources: [Allison Kaptur]() Topics: [Compiler](), [Optimization](), [Testing](), [test-coverage]() Tags: [compiler](), [compiler-optimization](), [python](), [test-coverage](), [testing]() ### AI overview This article explains a surprising side effect of the CPython peephole compiler optimization through a small Python test coverage tool built with sys.settrace. It demonstrates tracing executed lines while running a simple test framework and identifies lines that were not executed. ### Source excerpt Last Thursday I gave a lightning talk at Hacker School about the peephole optimizer in Python. A "peephole optimization" is a compiler optimization that looks at a small chunk of code at a time and optimizes in that little spot. This post explains one surprising side-effect of an optimization in CPython. Writing a test coverage tool Suppose that we're setting out to write a test coverage tool. Python provides an easy way to trace execution using sys.settrace, so a simple version of a coverage analyzer isn't too hard. Our code to test is one simple function: example.py 1 2 3 4 5 def iffer(condition): if condition: return 3 else: return 10 Then we'll write the world's simplest testing framework: tests.py 1 2 3 4 5 6 7 8 from example import iffer def test_iffer(): assert iffer(True) == 3 assert iffer(False) == 10 def run_tests(): test_iffer() Now for the simplest possible coverage tool. We can pass sys.settrace any tracing function, and it'll be called with the arguments frame, event, and arg every time an event happens in the execution. Lines of code being executed, function calls, function returns, and exceptions are all events. We'll filter out everything but line and call events, then keep track of what line of code was executing.1 Then we run the tests while the trace function is tracing, and finally report which (non-empty lines) failed to execute. coverage.py 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 import sys import tests import inspect class TinyCoverage(object): def __init__(self, file_to_watch): self.source_file = file_to_watch self.source_code = open(file_to_watch).readlines() self.executed_code = [] def trace(self, frame, event, arg): current_file = inspect.getframeinfo(frame).filename if self.source_file in current_file and \ (event == "line" or event == "call"): self.executed_code.append(frame.f_lineno) return self.trace def unexecuted_code(self): skipped = [] for line_n ## Of Syntax Warnings and Symbol Tables DevFeed: [Of Syntax Warnings and Symbol Tables]() Original publisher: [Read original article]() Published: 2014-06-12T01:30:00Z Content type: article Language: en Sources: [Allison Kaptur]() Topics: [Python](), [import](), [syntax](), [Code]() Tags: [bug](), [code](), [import](), [python](), [syntax]() ### AI overview The article examines Python's SyntaxWarning for star-imports outside module scope. It explains that the warning can occur without an immediate syntax error, while free variables in a function or nested block can cause the compiler to raise a SyntaxError. ### Source excerpt A Hacker Schooler hit an interesting bug today: her program would sometimes emit the message SyntaxWarning: import * only allowed at module level. I had never seen a SyntaxWarning before, so I decided to dig in. The wording of the warning is strange: it says that star-import is only allowed at the module level, but it's not a syntax error, just a warning. In fact, you can use a star-import in a scope that isn't a module (in Python 2): 1 2 3 4 5 6 7 >>> def nope(): ... from random import * ... print randint(1,10) ... :1: SyntaxWarning: import * only allowed at module level >>> nope() 7 The Python spec gives some more details: The from form with * may only occur in a module scope. If the wild card form of import -- import * -- is used in a function and the function contains or is a nested block with free variables, the compiler will raise a SyntaxError. Just having import * in a function isn't enough to raise a syntax error - we also need free variables. The Python execution model refers to three kinds of variables, 'local,' 'global,' and 'free', defined as follows: If a name is bound in a block, it is a local variable of that block. If a name is bound at the module level, it is a global variable. (The variables of the module code block are local and global.) If a variable is used in a code block but not defined there, it is a free variable. Now we can see how to trigger a syntax error from our syntax warning: 1 2 3 4 5 6 7 8 9 >>> def one(): ... def two(): ... from random import * ... print randint(1,10) ... two() ... :2: SyntaxWarning: import * only allowed at module level File "", line 3 SyntaxError: import * is not allowed in function 'two' because it is a nested function and similarly, 1 2 3 4 5 6 7 8 9 >>> def one(): ... from random import * ... def two(): ... print randint(1,10) ... two() ... :1: SyntaxWarning: import * only allowed at module level File "", line 2 SyntaxError: import * is not allowed in function 'one' because it ## PyCon prep: import is a keyword DevFeed: [PyCon prep: import is a keyword]() Original publisher: [Read original article]() Published: 2014-03-16T23:08:00Z Content type: opinion Language: en Sources: [Allison Kaptur]() Topics: [import](), [Python](), [Programming](), [function]() Tags: [function](), [import](), [programming](), [python]() ### AI overview The article examines why Python's import statement is a keyword and compares it with the __import__ function. It argues that the distinction is largely aesthetic and uses the topic to encourage reconsideration of familiar programming-language design decisions in a PyCon talk. ### Source excerpt Last week I had a ton of fun working with Amy Hanlon on her Harry Potter themed fork of Python, called Nagini. Nagini is full of magic and surprises. It implements the things you'd hope for out of a Harry Potter Python, like making quit into avada_kedavra, and many analogous jokes. Amy also had the idea to replace import with accio! Replacing import is a much harder problem than renaming a builtin. Python doesn't prevent you from overwriting builtins, whereas to change keywords you have to edit the grammar and recompile Python. You should go read Amy's post on making this work. This brings us to an interesting question: why is import a keyword, anyway? There's a function, __import__, that does (mostly) the same thing: 1 2 >>> __import__('random') The function form requires the programmer to assign the return value - the module - to a name, but once we've done that it works just like a normal module: 1 2 3 >>> random = __import__('random') >>> random.random() 0.32574174955668145 The __import__ function can handle all the forms of import, including from foo import bar and from baz import * (although it never modifies the calling namespace). There's no technical reason why __import__ couldn't be the regular way to do imports.1 As far as I can tell, the main argument against an import function is aesthetic. Compare: 1 2 3 4 5 6 7 import foo from foo import bar import longmodulename as short foo = __import__('foo') bar = __import__('random').bar short = __import__('longmodulename') The first way certainly feels much cleaner and more readable.2 Part of my goal in my upcoming PyCon talk is to invite Pythonistas to consider decisions they might not have thought about before. import is a great vehicle for this, because everyone learns it very early on in their programming development, but most people don't ever think about it again. Here's another variation on that theme: import doesn't have to be a keyword! I think all keywords c ## Reading EBNF DevFeed: [Reading EBNF]() Original publisher: [Read original article]() Published: 2014-03-16T22:39:00Z Content type: tutorial Language: en Sources: [Allison Kaptur]() Topics: [Parsing](), [syntax](), [Python](), [Parser]() Tags: [how-to](), [import](), [python](), [syntax]() ### AI overview A tutorial on reading Python's grammar in Extended Backus-Naur Form (EBNF), using import statements to explain how grammar rules form a tree and how literal keywords differ from named parser nodes. ### Source excerpt One of the fun parts of pairing with Amy on Nagini was modifying the grammar of Python. You should try this - it's easier than you think! Eli Bendersky has a great post with step-by-step instructions.1 Modifying Python's grammar starts in the Grammar/Grammar file. I've recently learned how to read this (which, for me, mostly means learning how to pronounce the punctuation), so I want to walk through the import example in some detail. The syntax here is Extended Backus-Naur Form, or EBNF. You read it like a tree, and your primary verb is "consists of": import_stmt consists of one of two forms, import_name or import_from. import_name consists of the literal word import followed by dotted_as_names. dotted_as_names consists of a dotted_as_name (note the singular), optionally followed by one or more pairs of a comma and another dotted_as_name. dotted_as_name consists of a dotted_name, optionally followed by the literal word 'as' and a NAME. Finally, dotted_name consists of a NAME, maybe followed by pairs of a dot and another NAME. You can walk the other branches in a similar way. 1 2 3 4 5 6 7 8 9 10 import_stmt: import_name | import_from import_name: 'import' dotted_as_names # note below: the ('.' | '...') is necessary because '...' is tokenized as ELLIPSIS import_from: ('from' (('.' | '...')* dotted_name | ('.' | '...')+) 'import' ('*' | '(' import_as_names ')' | import_as_names)) import_as_name: NAME ['as' NAME] dotted_as_name: dotted_name ['as' NAME] import_as_names: import_as_name (',' import_as_name)* [','] dotted_as_names: dotted_as_name (',' dotted_as_name)* dotted_name: NAME ('.' NAME)* To accio-ify Python, we had to replace the occurences of 'import' with 'accio'. There are only two - we were only interested in the literal string import, not all the other names. import_as_name and so on are just nodes in the tree, and only matter to the parser and compiler. Every other keyword and symbol that has special meaning to the Python parser also appears in Grammar as a ## PyCon prep: \`require\` in Ruby DevFeed: [PyCon prep: \`require\` in Ruby]() Original publisher: [Read original article]() Published: 2014-02-16T22:18:00Z Content type: article Language: en Sources: [Allison Kaptur]() Topics: [Ruby](), [import](), [modules](), [Python](), [Rails]() Tags: [file](), [import](), [modules](), [python](), [ruby]() ### AI overview A PyCon preparation note comparing Ruby's require and require_relative with Python's import. It explains file loading, file scope, module creation, namespace collisions, and the fact that required files are loaded only once. ### Source excerpt I'm talking about import at PyCon in April. In the talk, we'll imagine that there is no import and will reinvent it from scratch. I hope this will give everyone (including me!) a deeper understanding of the choices import makes and the ways it could have been different. Ideally, the structure will be a couple of sections of the form "We could have made [decisions]. That would mean [effects]. Surprise - that's how it works in [language]!"1 This is the first of (probably) several posts with notes of things I'm learning as I prepare my talk. Feedback is welcome. Today I'm looking into Ruby's require and require_relative2 to see if aspects of them would be interesting to Python programmers. So far, here's what I think is most relevant: Unlike Python, require won't load all objects in the required file. There's a concept of local versus global variables in the file scope that doesn't exist in Python. Unlike Python, one file does not map to one module. Modules are created by using the keyword module. Unlike Python, namespace collisions are completely possible. Consider the following simple files: one.rb 1 2 3 4 5 puts "one!" def foo :hello end two.rb 1 2 3 4 5 puts "two!" def foo :world end main.rb 1 2 3 4 require_relative 'one' require_relative 'two' puts foo And the output from running main.rb: output 1 2 3 one! two! world Like Python's import, require will only load a file once. This can interact interestingly with namespace collisions - to take a contrived example: main.rb 1 2 3 4 5 require_relative 'one' require_relative 'two' require_relative 'one' puts foo Because one.rb isn't reloaded, foo is still 'world': output 1 2 3 one! two! world Questions for further investigation / thought My talk should not convince people that Python is Right and other languages are Wrong. I'm trying to overcome my bias towards the system I'm most used to. (I think I've written roughly equal amounts of Python and Ruby, but the vast majority of the Ruby I've written is Rails, where all th ## How Python Compiles Source Code to Bytecode for Its Interpreter DevFeed: [How Python Compiles Source Code to Bytecode for Its Interpreter]() Original publisher: [Read original article]() Published: 2013-12-03T19:25:00Z Content type: article Language: en Sources: [Allison Kaptur]() Topics: [Python](), [Compiler](), [compiled language](), [Code](), [function]() Tags: [bytecode](), [compilation](), [compiled-language](), [compiler](), [python]() ### AI overview This fourth article in a series on the Python interpreter explains how Python can be dynamic while still being compiled. It describes Python's compilation of source code into bytecode for a virtual machine and illustrates the process with a function and its disassembled bytecode. ### Source excerpt [Edit: A significantly expanded version of this series appears as a chapter in The Architecture of Open Source Applications, volume 4, as A Python Interpreter Written in Python.] This is Part 4 in a series on the Python interpreter. Read Part 1, Part 2, and Part 3. If you're enjoying this series, consider applying to Hacker School, where I work as a facilitator. One of the things I was confused about when I started digging into python internals was how python could be "dynamic" if it was also "compiled." Often, in casual coversation, those two words are used as antonyms - there are "dynamic languages,"1 like Python, Ruby, and Javascript, and "compiled languages," like C, Java, and Haskell. Most of the time, when people talk about a "compiled" language, they mean one that compiles down to native x86/ARM/etc instructions2 - instructions for an actual machine made of metal. An "interpreted" language either doesn't have any compilation at all3, or compiles to an intermediate representation, like bytecode. Bytecode is instructions for a virtual machine, not a piece of hardware. Python falls into this latter category: the Python compiler's job is to generate bytecode for the Python interpreter.4 The Python interpreter's job is to make sense of the bytecode via the virtual machine, which turns out to be a lot of work. We'll dig in to the virtual machine in Part 5. So far our discussion of compiling versus interpretation has been abstract. These ideas become more clear with an example. 1 2 3 4 5 6 7 8 9 10 >>> def modulus(x, y): ... return x % y ... >>> [ord(b) for b in modulus.func_code.co_code] [124, 0, 0, 124, 1, 0, 22, 83] >>> dis.dis(modulus.func_code) 2 0 LOAD_FAST 0 (x) 3 LOAD_FAST 1 (y) 6 BINARY_MODULO 7 RETURN_VALUE Here's a function, its bytecode, and its bytecode run through the disassembler. By the time we get the prompt back after the function definition, the modulus function has been compiled and a code object generated. That code object will never be modified ## Introduction to the Python Interpreter, Part 3: Understanding Bytecode DevFeed: [Introduction to the Python Interpreter, Part 3: Understanding Bytecode]() Original publisher: [Read original article]() Published: 2013-11-17T17:56:00Z Content type: tutorial Language: en Sources: [Allison Kaptur]() Topics: [Python](), [Code]() Tags: [bytecode](), [code](), [internals](), [python]() ### AI overview Part 3 of a series on the Python interpreter explains that Python bytecode is a sequence of bytes stored in a code object's co_code attribute. It introduces disassembly with Python's dis module as a way to inspect this intermediate interpreter state. ### Source excerpt [Edit: A significantly expanded version of this series appears as a chapter in The Architecture of Open Source Applications, volume 4, as A Python Interpreter Written in Python.] This is Part 3 in a series on the Python interpreter. Part 1 here, Part 2 here. If you're enjoying this series, consider applying to Hacker School, where I work as a facilitator. Bytecode When we left our heroes, they had come across some odd-looking output: 1 2 >>> foo.func_code.co_code 'd\x01\x00}\x01\x00|\x01\x00|\x00\x00\x17S' This is python bytecode. You recall from Part 2 that "python bytecode" and "a python code object" are not the same thing: the bytecode is an attribute of the code object, among many other attributes. Bytecode is found in the co_code attribute of the code object, and contains instructions for the interpreter. So what is bytecode? Well, it's just a series of bytes. They look wacky when we print them because some bytes are printable and others aren't, so let's take the ord of each byte to see that they're just numbers. 1 2 >>> [ord(b) for b in foo.func_code.co_code] [100, 1, 0, 125, 1, 0, 124, 1, 0, 124, 0, 0, 23, 83] Here are the bytes that make up python bytecode. The interpreter will loop through each byte, look up what it should do for each one, and then do that thing. Notice that the bytecode itself doesn't include any python objects, or references to objects, or anything like that. One way to understand python bytecode would be to find the CPython interpreter file (it's ceval.c), and flip through it looking up what 100 means, then 1, then 0, and so on. We'll do this later in the series! For now, there's a simpler way: the dis module. Disassembling bytecode Disassembling bytecode means taking this series of bytes and printing out something we humans can understand. It's not a step in python execution; the dis module just helps us understand an intermediate state of python internals. I can't think of a reason why you'd ever want to use dis in production code - it ## Introduction to the Python Interpreter, Part 2: Code Objects DevFeed: [Introduction to the Python Interpreter, Part 2: Code Objects]() Original publisher: [Read original article]() Published: 2013-11-16T03:22:00Z Content type: tutorial Language: en Sources: [Allison Kaptur]() Topics: [Python](), [Code](), [Compiler](), [function]() Tags: [code](), [compiler](), [function](), [python]() ### AI overview This installment of a series on the Python interpreter examines function code objects. It explains that a code object is generated by the Python compiler and interpreted by the interpreter, then inspects attributes such as variable names, constants, and argument count. ### Source excerpt [Edit: A significantly expanded version of this series appears as a chapter in The Architecture of Open Source Applications, volume 4, as A Python Interpreter Written in Python.] This is part of a series on the python interpreter. Part 1 here. When we left our heroes, they were examining a simple function object. Let's now dive a level deeper, and look at this function's code object. 1 2 3 4 5 6 7 8 >>> def foo(a): ... x = 3 ... return x + a ... >>> foo >>> foo.func_code ", line 1> As you can see in the code above, the code object is an attribute of the function object. (There are lots of other attributes on the function object, too. They're mostly not interesting because foo is so simple.) A code object is generated by the Python compiler and intepreted by the interpreter. It contains information that this interpreter needs to do its job. Let's look at the attributes of the code object. 1 2 3 4 5 6 7 >>> dir(foo.func_code) ['__class__', '__cmp__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames'] There's a bunch of stuff going on here, much of which we're not going to worry about today. Let's take a look at three attributes that are interesting to us for our code object on foo. 1 2 3 4 5 6 >>> foo.func_code.co_varnames ('a', 'x') >>> foo.func_code.co_consts (None, 3) >>> foo.func_code.co_argcount 1 Here are some intelligible-looking things: the names of the variables and the constants that our function knows about and the number of arguments the function takes. But so far, we haven't seen anythi ## Introduction to the Python Interpreter, Part 1: Function Objects DevFeed: [Introduction to the Python Interpreter, Part 1: Function Objects]() Original publisher: [Read original article]() Published: 2013-11-16T02:50:00Z Content type: tutorial Language: en Sources: [Allison Kaptur]() Topics: [Python](), [Parsing](), [Code](), [Compiler]() Tags: [bytecode](), [code](), [compiler](), [parsing](), [python](), [syntax](), [tokens]() ### AI overview An introductory tutorial on Python interpreter internals. It explains how Python code is lexed, parsed into an abstract syntax tree, compiled into code objects, and interpreted, with the discussion focused on Python 2.7 and noting similarities with Python 3. ### Source excerpt [Edit: A significantly expanded version of this series appears as a chapter in The Architecture of Open Source Applications, volume 4, as A Python Interpreter Written in Python.] Over the last three months, I've spent a lot of time working with Ned Batchelder on byterun, a python bytecode interpreter written in python. Working on byterun has been tremendously educational and a lot of fun for me. At the end of this series, I'm going to attempt to convince you that it would be interesting and fun for you to play with byterun, too. But before we do that, we need a bit of a warm-up: an overview of how python's internals work, so that we can understand what an interpreter is, what it does, and what it doesn't do. This series assumes that you're in a similar position to where I was three months ago: you know python, but you don't know anything about the internals. One quick note: I'm going to work in and talk about Python 2.7 in this post. The interpreter in Python 3 is mostly pretty similar. There are also some syntax and naming differences, which I'm going to ignore, but everything we do here is possible in Python 3 as well. How does it python? We'll start out with a really (really) high-level view of python's internals. What happens when you execute a line of code in your python REPL? 1 2 3 4 5 ~ $ python Python 2.7.2 (default, Jun 20 2012, 16:23:33) [GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> a = "hello" There are four steps that python takes when you hit return: lexing, parsing, compiling, and interpreting. Lexing is breaking the line of code you just typed into tokens. The parser takes those tokens and generates a structure that shows their relationship to each other (in this case, an Abstract Syntax Tree). The compiler then takes the AST and turns it into one (or more) code objects. Finally, the interpreter takes each code object executes the code it represen ## Python Puzzle Solutions DevFeed: [Python Puzzle Solutions]() Original publisher: [Read original article]() Published: 2013-10-31T17:47:00Z Content type: article Language: en Sources: [Allison Kaptur]() Topics: [Python](), [test](), [function]() Tags: [blog-post](), [puzzle](), [python](), [solutions]() ### AI overview The article clarifies the rules for a Python puzzle and discusses submitted solutions, including Jessica McKellar's stateful solution and a correction that it passes the order test. ### Source excerpt I really enjoyed seeing all the clever solutions to the python puzzle I posted. You're all very creative! Here's a discussion of the solutions I've seen, plus some clarifications. All spoilers are below the fold. First, clarifications. (These weren't always clear in the problem statement, particularly if you got the problem off of twitter, so award yourself full marks as desired.) Order doesn't matter "Order doesn't matter" means that the three-line version always returns False, and the semicolon version always returns True. You control only the contents of the lines Several people, including Pepijn De Vos, David Wolever, and diarmuidbourke suggested something like the following: 1 2 3 4 5 6 >>> """a; b; c""" == 'a; b; c' True >>> """a ... b ... c""" == 'a; b; c' False I'm being pedantic here, but I rule this cheating, since (a) each line has to be a valid python expression or statement, and a multi-line string literal is only one expression, and (b) the string """a; b; c""" is not the same as the string """a\nb\nc""". Solutions appear below the fold. Solutions! Jessica McKellar Jessica suggests the following solution: 1 2 3 4 5 6 7 8 9 10 11 12 13 >>> global a >>> a = a + "a" if "a" in globals() else "" >>> print(bool(len(a) % 3)) False >>> global a; a = a + "a" if "a" in globals() else ""; print(bool(len(a) % 3)) True >>> def my_function(): ... global a ... a = a + "a" if "a" in globals() else "" ... print(bool(len(a) % 3)) ... >>> my_function() True However, Jessica's solution fails the "order doesn't matter" test, and it is stateful: 1 2 3 4 >>> global a; a = a + "a" if "a" in globals() else ""; print(bool(len(a) % 3)) False >>> global a; a = a + "a" if "a" in globals() else ""; print(bool(len(a) % 3)) True Edit: As Jessica points out, I'm wrong here: her solution does pass the order test. She also notes that the restriction against state wasn't present in the blog post (and she didn't see the original tweet). Full credit to Jessica, then! Javier Novoa Cataño Ja ## A Python puzzle DevFeed: [A Python puzzle]() Original publisher: [Read original article]() Published: 2013-10-30T02:40:00Z Content type: article Language: en Sources: [Allison Kaptur]() Topics: [Python](), [Code](), [function]() Tags: [code](), [function](), [puzzle](), [python](), [rules](), [stateful]() ### AI overview A Python puzzle challenges readers to find three valid Python expressions that produce different results when run separately, on one line with semicolons, and inside a function. The solution must avoid introspection, dunder methods, order dependence, and stateful behavior. ### Source excerpt A couple of Hacker Schoolers were discussing an interesting corner of python today. We discovered a nice bit of trivia: there exist three lines of python code that display the following behavior: 1 2 3 4 5 6 7 8 9 10 11 12 >>> LINE_A >>> LINE_B >>> LINE_C False >>> LINE_A; LINE_B; LINE_C True >>> def my_function(): ... LINE_A ... LINE_B ... LINE_C >>> my_function() True What are the lines? Some ground rules: Introspection of any kind is cheating (e.g. noting the line number). No dunder (__foo__) methods allowed. Each line is a valid python expression. You can't rely on order: while the lines will always execute A -> B -> C, a complete solution behaves identically if e.g. the semicolon version happens before the separate-line version. No cheating with the function: e.g. you can't add a return unless you add it everywhere. Edit: And nothing stateful. For bonus points, code golf! My solution to this is 14 19 characters long, not counting whitespace.