12/13/24, 5:34 PM Efficient way to write a list to file in python - Stack Overflow

Efficient way to write a list to file in python

Asked 5 years, 3 months ago Modified 5 years, 3 months ago Viewed 3k times

I'm trying to find the most efficient way to write a list to a file in python. My list is about 2
million items.
2
My first method is:

# first method

f = open("myfile.txt", "w")

mylist = [1,2,3, .... ,n]

for i in mylist:

#calculate a in function of i

f.write(str(a))

f.close()

My second method is:

# second method

list_to_write = []

mylist = [1,2,3, .... ,n]

for i in mylist:

#calculate a in function of i

list_to_write.append(a)

f = open("myfile2.txt", "w")
f.write(str(a))

And I have an alternative way to write the file with:

# third method (alternative way to write the file)

with open("myfile2.txt", 'w') as file:

file.write("\n".join(mylist))

The results are practically the same in all methods with one hundred items on my list. All
times are about a half minute with minor differences between them. The problem will be
when escalating to the two millions of items (and even more).

Is there another way to do it in a more efficient way?

https://stackoverflow.com/questions/57565758/efficient-way-to-write-a-list-to-file-in-python 1/3
12/13/24, 5:34 PM Efficient way to write a list to file in python - Stack Overflow

python list file

Share Improve this question edited Aug 20, 2019 at 2:02 asked Aug 20, 2019 at 1:50

Follow Fernando Barraza

137 1 2 14

I'm sorry in the second method is necessary to change i for a in the write method. – Fernando Barraza
Aug 20, 2019 at 2:01

I have no idea what's going on in method 2 or method 3. Perhaps you can clarify.
– President James K. Polk Aug 20, 2019 at 2:01

Have you tried using pandas? – Daniel Butler Aug 20, 2019 at 2:02

@JamesKPolk Does my last edition about the second method solve your doubt? – Fernando Barraza
Aug 20, 2019 at 2:04

not in the slightest. – President James K. Polk Aug 20, 2019 at 2:16

You might just use pickle, it's much easier than doing it by hand the way you're trying and it should be
fast. Try it and compare the speeds with your examples. – President James K. Polk Aug 20, 2019 at 2:18

Storing two million items in a list might take quite a bit of memory, depending how large each item is. I
think the first method is probably best. – John Gordon Aug 20, 2019 at 2:26

Good question. I wish it was adequately answered. I'm also wondering if your second method is best
than the first one. – m26a May 16, 2022 at 22:13

Does this answer your question? Writing to file multiple times vs Saving it in memory and writing once
– m26a May 16, 2022 at 22:17

3 Answers Sorted by: Highest score (default)

Maybe this:

1 import json
a = [1,2,3]
with open('test.txt', 'w') as f:
f.write(json.dumps(a))

Share Improve this answer Follow answered Aug 20, 2019 at 2:06
akiva
369 1 9

The pickle module is the easiest way to serialize python objects to/from storage. It is perfect
for python-python communications but not so good for communicating between python and
1 non-python systems. In that case, json is very simple and well-supported format.

Here is a simple code snippet showing pickle usage.

https://stackoverflow.com/questions/57565758/efficient-way-to-write-a-list-to-file-in-python 2/3
12/13/24, 5:34 PM Efficient way to write a list to file in python - Stack Overflow

# this is saved in file "scratch.py"

import pickle
import random

r = random.Random()

l = [r.getrandbits(31) for _ in range(2_000_000)]

with open('saved.pkl', 'wb') as f:

pickle.dump(l, f, pickle.HIGHEST_PROTOCOL)

timing and storage on my macbook pro circa 2017:

$ time python3 scratch.py

real 0m0.571s
user 0m0.512s
sys 0m0.053s

$ ls -hl saved.pkl
-rw-r--r-- 1 jameskpolk staff 9.5M Aug 20 17:49 saved.pkl

Share Improve this answer Follow answered Aug 20, 2019 at 21:57
President James K. Polk
41.9k 26 107 141

you can first convert the list into pandas Series and use the to_csv method to write it to disk.

-1 import pandas as pd
mylist = [1,2,3,...,n]
pd.Series(mylist).to_csv('myfile.txt', header=False, index=False)

Share Improve this answer Follow edited Aug 20, 2019 at 2:27 answered Aug 20, 2019 at 2:09
haijohn
59 4

https://stackoverflow.com/questions/57565758/efficient-way-to-write-a-list-to-file-in-python 3/3