How do I read a huge file line by line in Python?

by

This is taken from here and was written by rupe.

In Python, the most common way to read lines from a file is to do the following:

for line in open('myfile','r').readlines():
do_something(line)

When this is done, however, the readlines() function loads the entire file into memory as it runs. A better approach for large files is to use the fileinput module, as follows:

import fileinput
for line in fileinput.input(['myfile']):
do_something(line)

the fileinput.input() call reads lines sequentially, but doesn’t keep them in memory after they’ve been read.

Advertisements

5 Responses to “How do I read a huge file line by line in Python?”

  1. Ettore Says:

    Thanks Chris, this could help me doing something I need at the moment 😉 Very timely!

  2. della Says:

    for line in open(‘myfile’): blah() does exactly the same thing: file objects can be iterated from python 2.3.

    Even better, starting from py 2.5, you can ensure that the file object is closed as soon as you exit from the block using it with the ‘with’ statement.

    with open(‘myfile’) as f:
    for line in f:
    blah()

    Of course there would be no problem in calling f.close(), but this ensures that the object gets closed however you exit from the block (exception, ‘break’ statement, etc.

  3. Sarah Dunne Says:

    Its like you read my mind!! You appear to know so much about this, like you wrote the book in it or something.. I think that you could do with some pics to drive the message home a bit, but other than that, this is wonderful blog. A fantastic read… I’ll definitely be back. Irvine Roofing Contractors, 18242 McDurmott W., #G-2, Irvine, CA, 92614, US, 949-390-9929

  4. www.youtube.com Says:

    Hiya very cool website!! Guy .. Beautiful .. Amazing .
    . I’ll bookmark your website and take the feeds additionally? I am glad to seek out so many useful info right here within the post, we need develop more strategies on this regard, thanks for sharing. . . . . .

  5. Fort Worth Roofing roofing contractor website Says:

    Admiring the commitment you put into your
    site and in depth information you provide. It’s great to come across a blog every once in a while that isn’t the same old rehashed information.
    Wonderful read! I’ve saved your site and I’m including your RSS feeds to my Google account.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: