Python • File I/O & Context Managers

File I/O: Read, Write & Context Managers

Master Python file operations – open, read, write, append, context managers, CSV, JSON, binary files, and production‑grade file‑handling patterns with detailed examples, solved problems, and practice challenges.

Topics
1
2
3
4
5
6
7

📌 What you'll learn: File I/O is essential for reading configuration, processing data files, logging, and persisting information. Python makes file handling simple yet powerful with built‑in functions, the with statement, and libraries like csv, json, and pathlib. This page covers everything from basic open() to production‑safe file operations.

🎯 Goal: Write safe, efficient, and idiomatic file‑handling code — whether you're parsing a small text file or building a data pipeline that processes thousands of files reliably.

01

File Basics

open()close()
02

File Modes

r/w/ar+/b
03

Context Managers

with__enter__
04

Reading

read()readlines()
05

Writing

write()writelines()
06

CSV & JSON

csvjson
07

Production

pathlibtempfile

Why File I/O & Context Managers Matter

What problems do they solve?

File I/O allows programs to persist data beyond runtime, read configuration, process datasets, and communicate with other systems. Context managers guarantee resources are released properly — even when errors occur — preventing file corruption and resource leaks.

Why are they required?

Without proper file handling, you risk data loss, file corruption, and exhausting system file descriptors. Manually calling close() is error‑prone; forgetting it can lock files and crash long‑running applications.

Role in Production

Production systems read configs, write logs, process uploads, and manage temporary files. Using pathlib for cross‑platform paths, tempfile for secure temporary files, and atomic writes ensures reliability at scale.

1 File Basics – open(), read(), write(), close()

The open() function returns a file object. Always close files to free system resources. The default mode is 'r' (read text). Use 'w' to write (overwrites existing content) and 'a' to append.

file_basics.py
# Writing to a file
file = open("hello.txt", "w")
file.write("Hello, World!\n")
file.write("This is line 2.\n")
file.close()  # ⚠️ Don't forget to close!

# Reading from a file
file = open("hello.txt", "r")
content = file.read()       # reads entire file as a single string
print(content)
file.close()

# Appending to a file
file = open("hello.txt", "a")
file.write("Appended line.\n")
file.close()
🔍 Key Insight: The file object is a stream. After reading, the cursor moves to the end. Use file.seek(0) to reset to the beginning if you need to re‑read.

2 File Modes Deep Dive

Python supports multiple modes. Combine read/write with text/binary. Here's a comprehensive breakdown:

ModeMeaningFile ExistsFile Doesn't Exist
'r'Read (text, default)Opens for reading❌ Raises FileNotFoundError
'w'Write (text)⚠️ Truncates (overwrites)Creates new file
'a'Append (text)Appends at endCreates new file
'x'Exclusive creation❌ Raises FileExistsErrorCreates new file
'r+'Read + WriteOpens for both; no truncation❌ Raises error
'w+'Write + Read⚠️ Truncates firstCreates new file
'rb'Read binaryOpens in binary mode❌ Raises error
'wb'Write binary⚠️ TruncatesCreates new file
file_modes_demo.py
# Exclusive creation – safe when you don't want to overwrite
try:
    with open("unique_file.txt", "x") as f:
        f.write("Created exclusively!\n")
except FileExistsError:
    print("File already exists – skipping.")

# Binary mode – for images, PDFs, etc.
with open("image_copy.png", "wb") as dest:
    with open("original.png", "rb") as src:
        dest.write(src.read())  # copies binary data
⚠️ Warning: 'w' mode silently deletes existing content. Use 'x' if you want to avoid accidental overwrites, or check with os.path.exists() first.

3 Context Managers – The with Statement

The with statement ensures files are closed automatically — even if an exception occurs. This is the idiomatic Python way to handle files. You can also build custom context managers using classes or the contextlib module.

context_manager_demo.py
# ✅ The Pythonic way – no explicit close() needed
with open("data.txt", "r") as file:
    content = file.read()
    print(content)
# File is automatically closed here, even if an error occurred

# Custom context manager using a class
class ManagedFile:
    def __init__(self, filename, mode='r'):
        self.filename = filename
        self.mode = mode

    def __enter__(self):
        self.file = open(self.filename, self.mode)
        return self.file

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.file.close()
        if exc_type is not None:
            print(f"An error occurred: {exc_val}")
        return False  # don't suppress exceptions

with ManagedFile("log.txt", "w") as f:
    f.write("Logging with custom context manager!\n")
💡 Pro Tip: Use from contextlib import contextmanager with the @contextmanager decorator and yield for simpler generator‑based context managers when you don't need a full class.
🧠 How with works step‑by‑step:
  1. open() is called → returns a file object.
  2. The file object's __enter__() method runs → returns the file handle.
  3. The block inside with executes.
  4. When the block ends (or an exception occurs), __exit__() is called → file is closed.

4 Reading Techniques – read(), readline(), readlines(), Iteration

Choose the right reading method based on file size and your use case. For large files, iterate line‑by‑line to avoid loading everything into memory.

reading_methods.py
# 1. read() – entire file as one string (use for small files)
with open("poem.txt", "r") as f:
    whole_text = f.read()

# 2. readline() – one line at a time
with open("poem.txt", "r") as f:
    first_line = f.readline()
    second_line = f.readline()

# 3. readlines() – list of all lines (includes \n)
with open("poem.txt", "r") as f:
    all_lines = f.readlines()  # ['line1\n', 'line2\n', ...]

# 4. ✅ Best for large files – iterate directly over file object
with open("huge_log.txt", "r") as f:
    for line in f:
        process(line.strip())  # strip() removes \n and whitespace

# 5. read(n) – read n characters (or bytes in binary mode)
with open("data.txt", "r") as f:
    chunk = f.read(1024)  # read first 1024 characters
🔍 Memory Tip: f.read() on a 2 GB file will crash your program with MemoryError. Always iterate with for line in f for files larger than available RAM.

5 Writing Techniques – write(), writelines(), Flushing & Buffering

Writing is straightforward, but understanding buffering and flushing helps when you need real‑time output (e.g., logging to a file that another process reads).

writing_methods.py
# 1. write() – writes a single string
with open("output.txt", "w") as f:
    f.write("Line 1\n")
    f.write("Line 2\n")

# 2. writelines() – writes an iterable of strings
lines = ["apple\n", "banana\n", "cherry\n"]
with open("fruits.txt", "w") as f:
    f.writelines(lines)

# 3. print() to file – convenient for formatted output
data = {"name": "Alice", "score": 95}
with open("report.txt", "w") as f:
    print(f"Student: {data['name']}", file=f)
    print(f"Score: {data['score']}", file=f)

# 4. Flushing – force write to disk immediately
with open("live_log.txt", "w", buffering=1) as f:  # line-buffered
    f.write("Processing...\n")
    f.flush()  # ensures it's written now (useful for tail -f)
💡 Buffering modes: buffering=0 (unbuffered, binary only), buffering=1 (line‑buffered, text only), buffering>1 (fixed buffer size in bytes). Default is system‑dependent (~8KB).

6 Working with CSV & JSON Files

Python's csv module handles delimiter parsing, quoting, and dialect detection. The json module serializes Python objects to JSON strings and deserializes them back — perfect for configs, APIs, and data exchange.

csv_json_demo.py
import csv
import json

# ========== CSV READING ==========
with open("users.csv", "r", newline='') as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(f"{row['name']} – {row['email']}")

# ========== CSV WRITING ==========
with open("export.csv", "w", newline='') as f:
    writer = csv.DictWriter(f, fieldnames=["name", "score"])
    writer.writeheader()
    writer.writerow({"name": "Alice", "score": 95})
    writer.writerow({"name": "Bob", "score": 87})

# ========== JSON READING ==========
with open("config.json", "r") as f:
    config = json.load(f)   # returns dict/list
    print(config["api_key"])

# ========== JSON WRITING ==========
data = {"users": [{"name": "Alice"}, {"name": "Bob"}]}
with open("output.json", "w") as f:
    json.dump(data, f, indent=2, ensure_ascii=False)
🔍 Key Insight: Always use newline='' when opening CSV files — the csv module handles newlines itself, and not doing so can cause extra blank lines on Windows.

7 Production‑Grade File I/O Patterns

In production, use pathlib for elegant path manipulation, tempfile for secure temporary files, atomic writes to prevent corruption, and proper encoding handling.

production_patterns.py
from pathlib import Path
import tempfile
import os

# ========== PATHLIB – modern path handling ==========
data_dir = Path("data") / "2025" / "reports"
data_dir.mkdir(parents=True, exist_ok=True)  # create dirs if needed
report_path = data_dir / "summary.txt"
report_path.write_text("Q1 Report\n", encoding="utf-8")
content = report_path.read_text(encoding="utf-8")

# ========== TEMPFILE – secure temporary files ==========
with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as tmp:
    tmp.write("Temporary data\n")
    tmp_path = tmp.name  # get the path
print(f"Temp file at: {tmp_path}")
# Process the file, then clean up
os.unlink(tmp_path)

# ========== ATOMIC WRITE – prevent partial writes ==========
def atomic_write(filepath, content):
    """Write to a temp file first, then rename atomically."""
    tmp_path = filepath + ".tmp"
    with open(tmp_path, "w", encoding="utf-8") as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())  # force write to disk
    os.replace(tmp_path, filepath)  # atomic rename (Unix/Windows)

atomic_write("important.txt", "Critical data here.\n")

# ========== ENCODING – always specify ==========
with open("unicode_file.txt", "w", encoding="utf-8") as f:
    f.write("Café résumé – 日本語もOK\n")
💡 Best Practice: Always specify encoding="utf-8" explicitly. The system default varies by OS (Windows uses cp1252, not UTF‑8), leading to bugs that only appear on certain machines.
⚠️ Atomic writes matter: If your program crashes mid‑write, a partially‑written file can corrupt downstream processes. Writing to a temp file + atomic rename guarantees either the old file or the complete new file — never a half‑written mess.

Error Handling in File Operations

File operations can fail for many reasons: missing files, permission errors, disk full, or encoding issues. Robust code handles these gracefully.

error_handling.py
def safe_read_file(filepath):
    """Read a file safely, handling common errors."""
    try:
        with open(filepath, "r", encoding="utf-8") as f:
            return f.read()
    except FileNotFoundError:
        print(f"❌ File not found: {filepath}")
        return None
    except PermissionError:
        print(f"❌ Permission denied: {filepath}")
        return None
    except UnicodeDecodeError:
        # Try with a different encoding
        with open(filepath, "r", encoding="latin-1") as f:
            return f.read()
    except OSError as e:
        print(f"❌ OS error: {e}")
        return None

content = safe_read_file("maybe_missing.txt")

Solved Practice Problems

🔰 Easy (3 problems)

1. Write and Read a Greeting File
with open("greeting.txt", "w") as f:
    f.write("Hello from Python!\n")
with open("greeting.txt", "r") as f:
    print(f.read())  # Hello from Python!
2. Count Lines in a File
def count_lines(filename):
    with open(filename, "r") as f:
        return sum(1 for _ in f)
print(count_lines("poem.txt"))
3. Append Timestamp to Log
from datetime import datetime
with open("app.log", "a") as f:
    f.write(f"[{datetime.now()}] App started.\n")

⚡ Intermediate (3 problems)

4. Copy File in Chunks (Memory‑Safe)
def copy_file(src, dest, chunk_size=8192):
    with open(src, "rb") as s, open(dest, "wb") as d:
        while chunk := s.read(chunk_size):
            d.write(chunk)
copy_file("large.mp4", "large_copy.mp4")
5. Read CSV and Filter Rows
import csv
with open("data.csv", "r", newline='') as f:
    reader = csv.DictReader(f)
    high_scorers = [row for row in reader if int(row['score']) > 80]
print(high_scorers)
6. JSON Config Loader with Defaults
import json
def load_config(path, defaults=None):
    try:
        with open(path, "r") as f:
            config = json.load(f)
        return {**defaults, **config} if defaults else config
    except (FileNotFoundError, json.JSONDecodeError):
        return defaults or {}
cfg = load_config("config.json", {"port": 8080})

🚀 Advanced (3 problems)

7. Custom Context Manager for Database File
class DatabaseFile:
    def __init__(self, path):
        self.path = path
    def __enter__(self):
        self.conn = open(self.path, "r+")
        return self.conn
    def __exit__(self, *args):
        self.conn.flush()
        self.conn.close()
with DatabaseFile("db.txt") as db:
    db.write("UPDATE users SET active=1;\n")
8. Atomic JSON Writer
import json, os
def atomic_json_write(path, data):
    tmp = path + ".tmp"
    with open(tmp, "w", encoding="utf-8") as f:
        json.dump(data, f, indent=2)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)
atomic_json_write("state.json", {"count": 42})
9. Recursive Directory File Counter with pathlib
from pathlib import Path
def count_files_by_extension(root_dir):
    counts = {}
    for file in Path(root_dir).rglob("*"):
        if file.is_file():
            ext = file.suffix or "(no extension)"
            counts[ext] = counts.get(ext, 0) + 1
    return counts
print(count_files_by_extension("./project"))

Unsolved Practice Problems

🔰 Easy (3 problems)

1. Write a Shopping List to a File

Write a function save_shopping_list(filename, items) that takes a list of strings and writes each item on a new line. Then write load_shopping_list(filename) that reads them back into a list.

2. Count Words in a Text File

Write a function word_count(filename) that returns the total number of words in a text file. Treat any whitespace‑separated sequence as a word.

3. File Exists Check Before Writing

Write a function safe_write(filename, content) that checks if the file exists. If it does, ask for confirmation (simulate with a boolean parameter overwrite). Use 'x' mode for safe creation.

⚡ Intermediate (3 problems)

4. Merge Multiple Text Files

Write merge_files(output, *input_files) that concatenates the contents of all input files into the output file. Use chunked reading for memory safety.

5. CSV to JSON Converter

Write a script that reads a CSV file (with headers) and converts it to a JSON array of objects. Handle missing values gracefully with a default.

6. Logging Context Manager

Create a context manager LogBlock(filename, message) that writes a start message when entering and an end message (with duration) when exiting. Use time.time() to measure elapsed time.

🚀 Advanced (3 problems)

7. File Watcher with Polling

Write a function watch_file(filename, interval=1.0) that polls the file's modification time (os.path.getmtime) and yields the new content whenever the file changes. Use an infinite loop with time.sleep.

8. Rotating File Writer

Create a class RotatingFileWriter that writes to a file but creates a new file (with a counter suffix) when the current file exceeds a given size limit (e.g., 1 MB).

9. Concurrent‑Safe File Append

Research and implement a function that safely appends to a file from multiple processes using fcntl.flock (Unix) or msvcrt.locking (Windows) for file locking. Explain why mode='a' alone isn't always safe for multi‑process writes.