Unbuffered output to stdout from Python programs

I recently had cause to debug a Python program that extracts data from a remote (MySQL) database and stores in a local SQLite database. I did the usual thing of inserting print statements to check a handful of variables at different stages of the process, but it wasn’t helping because the program would stay obstinately quiet until the end and then print all the information at once.

Wait, but why? Then I remembered that the info being written to stdout was buffered, so that I was only seeing the messages once the buffer had been filled. That’s all well and good, and perfectly efficient and all, but it was getting in the way of my debugging.

The solution I found was to configure stdout in the Python code to be unbuffered. Thanks to this helpful blog entry at Turnkey Linux, I added the following line to the start of my program (plus the required imports):

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

Bingo! No more waiting for my print statements to show up. Cue happy dance. Now I can get on with solving the actual problem I’m having with the code…

(I tried the other solution mentioned, stdbuf, but either I was using it wrong, or it wasn’t doing what I expected it do to.)

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s