Unbuffered output to stdout from Python programs

I recently had cause to debug a Python program that extracts data from a remote (MySQL) database and stores in a local SQLite database. I did the usual thing of inserting print statements to check a handful of variables at different stages of the process, but it wasn’t helping because the program would stay obstinately quiet until the end and then print all the information at once.

Wait, but why? Then I remembered that the info being written to stdout was buffered, so that I was only seeing the messages once the buffer had been filled. That’s all well and good, and perfectly efficient and all, but it was getting in the way of my debugging.

The solution I found was to configure stdout in the Python code to be unbuffered. Thanks to this helpful blog entry at Turnkey Linux, I added the following line to the start of my program (plus the required imports):

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

Bingo! No more waiting for my print statements to show up. Cue happy dance. Now I can get on with solving the actual problem I’m having with the code…

(I tried the other solution mentioned, stdbuf, but either I was using it wrong, or it wasn’t doing what I expected it do to.)

Advertisements