📦 Lesson 9: Virtual Environments & Package Management

💥 The Dependency Problem

Imagine you're working on two projects:

• Project A requires requests==2.28.0 (an older version)
• Project B requires requests==2.31.0 (the latest version)

If both projects share the same Python installation, you can only have one version of requests installed at a time. Upgrading for Project B breaks Project A. Downgrading for Project A breaks Project B. This is called dependency conflict, and it's one of the most common headaches in software development.

The solution? Give each project its own isolated Python environment with its own set of packages. That's exactly what virtual environments do.

🏗️ Creating Virtual Environments

Python 3 includes the venv module in its standard library — no extra installation needed.

Create a Virtual Environment

# Navigate to your project directory
# cd my_project

# Create a virtual environment named "venv" (convention)
# python -m venv venv

# Or name it anything you want
# python -m venv .venv
# python -m venv my_env

This creates a new directory (e.g., venv/) containing:

# venv/
# ├── bin/            (Linux/macOS) or Scripts/ (Windows)
# │   ├── python      → isolated Python binary
# │   ├── pip         → isolated pip
# │   └── activate    → activation script
# ├── lib/
# │   └── pythonX.Y/
# │       └── site-packages/   → where installed packages go
# ├── include/
# └── pyvenv.cfg      → configuration file

Activate the Virtual Environment

Activating a venv modifies your shell's PATH so that python and pip point to the venv's versions:

# Linux / macOS
# source venv/bin/activate

# Windows (Command Prompt)
# venv\Scripts\activate.bat

# Windows (PowerShell)
# venv\Scripts\Activate.ps1

# Windows (Git Bash)
# source venv/Scripts/activate

When activated, your terminal prompt changes to show the venv name:

# Before activation:
# $ python --version
# Python 3.12.0   ← system Python

# After activation:
# (venv) $ python --version
# Python 3.12.0   ← venv Python (same version, isolated packages)

# Verify which python is being used:
# (venv) $ which python     (Linux/macOS)
# /home/user/my_project/venv/bin/python

# (venv) $ where python     (Windows)
# C:\Users\user\my_project\venv\Scripts\python.exe

Deactivate

# Return to system Python
# (venv) $ deactivate
# $   ← back to normal prompt

# .gitignore
venv/
.venv/
__pycache__/
*.pyc

📥 Installing Packages with pip

pip is Python's package installer. With your venv activated, any packages you install go into the venv — not the system Python.

Basic Installation

# Install the latest version
# pip install requests

# Install a specific version
# pip install requests==2.31.0

# Install minimum version
# pip install requests>=2.28.0

# Install compatible version (same major.minor)
# pip install requests~=2.31.0   (allows 2.31.x but not 2.32.0)

# Install multiple packages at once
# pip install requests flask pandas

Managing Installed Packages

# List all installed packages
# pip list

# Show details about a specific package
# pip show requests
# Name: requests
# Version: 2.31.0
# Summary: Python HTTP for Humans.
# Requires: certifi, charset-normalizer, idna, urllib3
# Required-by: (nothing — no other installed package depends on it)

# Upgrade a package to latest
# pip install --upgrade requests

# Uninstall a package
# pip uninstall requests

# Check for outdated packages
# pip list --outdated

Installing from Different Sources

# From a requirements file
# pip install -r requirements.txt

# From a Git repository
# pip install git+https://github.com/user/repo.git

# From a local directory (editable/development mode)
# pip install -e .
# pip install -e ./my_package

# From a wheel file
# pip install my_package-1.0.0-py3-none-any.whl

📋 Managing Dependencies

A requirements.txt file is the standard way to declare and share your project's dependencies. It lists every package your project needs, with exact version numbers, so anyone can recreate the same environment.

Creating requirements.txt

# Freeze current environment to a file
# pip freeze > requirements.txt

# This produces something like:
# certifi==2023.7.22
# charset-normalizer==3.2.0
# idna==3.4
# requests==2.31.0
# urllib3==2.0.4

Recreating an Environment

# On a new machine or after cloning a repo:

# 1. Create a fresh virtual environment
# python -m venv venv

# 2. Activate it
# source venv/bin/activate   (Linux/macOS)
# venv\Scripts\activate      (Windows)

# 3. Install all dependencies from requirements.txt
# pip install -r requirements.txt

# Done! Your environment matches the original exactly.

Writing requirements.txt by Hand

You can also write requirements.txt manually for cleaner control. Use comments and flexible version specifiers:

# requirements.txt — My Awesome Project
# Core dependencies
requests>=2.28.0,<3.0.0
flask~=2.3.0
sqlalchemy>=2.0

# Data processing
pandas>=2.0
numpy>=1.24

# Development only (consider a separate dev-requirements.txt)
# pytest>=7.0
# black>=23.0
# mypy>=1.0

Splitting Dev vs. Production

# requirements.txt — production dependencies
requests>=2.28
flask~=2.3
gunicorn>=21.0

# dev-requirements.txt — development/testing tools
-r requirements.txt    # Include production deps
pytest>=7.0
pytest-cov>=4.0
black>=23.0
mypy>=1.0
flake8>=6.0

# Install production deps only (deployment)
# pip install -r requirements.txt

# Install everything including dev tools (development)
# pip install -r dev-requirements.txt

🧩 Python Modules

A module is simply a .py file. When you write import mymodule, Python looks for mymodule.py and executes its code, making its functions, classes, and variables available to you.

Creating and Importing a Module

# ── helpers.py ──
"""Utility functions for data processing."""

def clean_text(text):
    """Remove extra whitespace and lowercase."""
    return " ".join(text.lower().split())

def validate_email(email):
    """Basic email validation."""
    return "@" in email and "." in email.split("@")[-1]

PI = 3.14159

# ── main.py ──
# Import the entire module
import helpers

cleaned = helpers.clean_text("  Hello   World  ")
print(cleaned)  # "hello world"
print(helpers.PI)  # 3.14159

# Import specific items
from helpers import clean_text, validate_email

cleaned = clean_text("  Hello   World  ")
is_valid = validate_email("user@example.com")

# Import with alias
import helpers as h
cleaned = h.clean_text("Hello")

# Import everything (avoid in production — pollutes namespace)
from helpers import *

Where Python Looks for Modules

When you write import mymodule, Python searches these locations in order:

import sys
print(sys.path)
# [
#   '',                              # 1. Current directory
#   '/usr/lib/python3.12',           # 2. Standard library
#   '/usr/lib/python3.12/lib-dynload',
#   '/home/user/project/venv/lib/python3.12/site-packages',  # 3. venv packages
# ]

📂 Python Packages

A package is a directory containing Python modules and a special __init__.py file. Packages let you organize related modules into a hierarchy.

Basic Package Structure

# my_package/
# ├── __init__.py       ← Makes this directory a package
# ├── core.py           ← Module: my_package.core
# ├── utils.py          ← Module: my_package.utils
# └── models/           ← Sub-package
#     ├── __init__.py
#     ├── user.py        ← Module: my_package.models.user
#     └── product.py     ← Module: my_package.models.product

The __init__.py File

# ── my_package/__init__.py ──
"""My awesome package for data processing."""

# This file runs when someone imports the package.
# Use it to define the public API.

# Re-export commonly used items for convenient access
from .core import process_data, analyze
from .utils import clean_text

# Package-level constants
__version__ = "1.0.0"
__author__ = "Your Name"

# Now users can do:
# from my_package import process_data, clean_text
# instead of:
# from my_package.core import process_data

Importing from Packages

# Import the package (runs __init__.py)
import my_package
my_package.process_data(data)

# Import a specific module
from my_package import utils
utils.clean_text("hello")

# Import a specific function from a module
from my_package.models.user import User
user = User("Alice")

# Relative imports (from within the package)
# ── my_package/core.py ──
from .utils import clean_text        # Same level (.)
from .models.user import User         # Sub-package
from ..other_package import helper    # Parent level (..) — less common

🏗️ Project Structure

A well-organized project makes your code easier to navigate, test, and share. Here's the standard layout that the Python community has converged on:

Small Script Project

# my_script/
# ├── main.py              ← Entry point
# ├── helpers.py            ← Utility functions
# ├── requirements.txt      ← Dependencies
# ├── .gitignore
# └── README.md

Application Project

# my_app/
# ├── src/
# │   └── my_app/           ← Main package
# │       ├── __init__.py
# │       ├── __main__.py    ← Enables: python -m my_app
# │       ├── cli.py         ← Command-line interface
# │       ├── core.py        ← Business logic
# │       ├── models.py      ← Data models
# │       ├── utils.py       ← Utilities
# │       └── config.py      ← Configuration
# ├── tests/
# │   ├── __init__.py
# │   ├── test_core.py
# │   ├── test_models.py
# │   └── test_utils.py
# ├── data/                  ← Sample data, fixtures
# ├── docs/                  ← Documentation
# ├── requirements.txt
# ├── dev-requirements.txt
# ├── pyproject.toml         ← Modern project metadata
# ├── .gitignore
# └── README.md

Key Files Explained

A Practical .gitignore

# Virtual environments
venv/
.venv/
env/

# Python cache
__pycache__/
*.py[cod]
*$py.class
*.egg-info/
dist/
build/

# IDE settings
.vscode/
.idea/
*.swp

# Environment variables
.env

# OS files
.DS_Store
Thumbs.db

🎯 if __name__ == "__main__"

This is one of Python's most important idioms. Every Python file has a built-in variable called __name__. Its value depends on how the file is used:

• Run directly (python myfile.py): __name__ is "__main__"
• Imported (import myfile): __name__ is "myfile"

# ── math_tools.py ──
"""A collection of math utility functions."""

def factorial(n):
    """Compute n! iteratively."""
    result = 1
    for i in range(2, n + 1):
        result *= i
    return result

def is_prime(n):
    """Check if n is a prime number."""
    if n < 2:
        return False
    for i in range(2, int(n ** 0.5) + 1):
        if n % i == 0:
            return False
    return True


# This block ONLY runs when the file is executed directly
if __name__ == "__main__":
    # Test the functions
    print(f"5! = {factorial(5)}")        # 120
    print(f"7 is prime: {is_prime(7)}")  # True
    print(f"10 is prime: {is_prime(10)}")# False

    # Run a demo
    primes = [n for n in range(2, 50) if is_prime(n)]
    print(f"Primes under 50: {primes}")

# ── main.py ──
# When imported, only the functions are available — the test code doesn't run
from math_tools import factorial, is_prime

print(factorial(10))   # 3628800
print(is_prime(97))    # True
# The demo/test block in math_tools.py did NOT execute!

Common Patterns

# Pattern 1: Simple script with a main() function
def main():
    """Application entry point."""
    print("Starting application...")
    # ... your code here ...

if __name__ == "__main__":
    main()

# Pattern 2: CLI with argument parsing
import argparse

def main():
    parser = argparse.ArgumentParser(description="Process data files")
    parser.add_argument("input", help="Input file path")
    parser.add_argument("-o", "--output", default="output.csv", help="Output file")
    parser.add_argument("-v", "--verbose", action="store_true")
    args = parser.parse_args()

    # ... process args.input ...

if __name__ == "__main__":
    main()

# Pattern 3: __main__.py for package execution
# ── my_app/__main__.py ──
"""Allow running the package with: python -m my_app"""
from .cli import main
main()

🚀 Beyond the Basics

The venv + pip + requirements.txt workflow covers most needs. Here's a quick tour of tools you'll encounter as projects grow.

pyproject.toml — The Modern Standard

pyproject.toml (PEP 621) is the modern way to declare project metadata. It replaces the older setup.py and setup.cfg files:

# pyproject.toml
[project]
name = "my-app"
version = "1.0.0"
description = "A useful application"
requires-python = ">=3.10"
dependencies = [
    "requests>=2.28",
    "flask~=2.3",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "black>=23.0",
]

[project.scripts]
my-app = "my_app.cli:main"

Alternative Tools

🏋️ Hands-on Exercises

# Terminal commands (Linux/macOS — adapt for Windows):

# 1. Create project directory
# mkdir weather_app
# cd weather_app

# 2. Create virtual environment
# python -m venv venv

# 3. Activate it
# source venv/bin/activate
# (venv) $   ← prompt should change

# 4. Install packages
# pip install requests python-dotenv

# 5. Freeze dependencies
# pip freeze > requirements.txt
# cat requirements.txt
# certifi==2023.7.22
# charset-normalizer==3.3.2
# idna==3.6
# python-dotenv==1.0.0
# requests==2.31.0
# urllib3==2.1.0

# 6. Create .gitignore
# Create a file named .gitignore with this content:
# venv/
# __pycache__/
# *.pyc
# .env

# 7. Verify
# deactivate
# source venv/bin/activate
# pip list
# Package            Version
# certifi            2023.7.22
# charset-normalizer 3.3.2
# idna               3.6
# pip                23.3.1
# python-dotenv      1.0.0
# requests           2.31.0
# urllib3             2.1.0

# ✅ All packages intact after reactivation!

# ── textkit/__init__.py ──
"""textkit — A lightweight text processing toolkit."""

from .cleaners import strip_html, normalize_whitespace
from .analyzers import word_count, char_frequency
from .formatters import title_case, truncate

__version__ = "1.0.0"

# ── textkit/cleaners.py ──
"""Text cleaning utilities."""
import re

def strip_html(text):
    """Remove HTML tags from text."""
    return re.sub(r"<[^>]+>", "", text)

def normalize_whitespace(text):
    """Collapse multiple whitespace characters into single spaces."""
    return " ".join(text.split())

if __name__ == "__main__":
    html = "<h1>Hello</h1> <p>World</p>"
    print(f"strip_html: '{strip_html(html)}'")
    # strip_html: 'Hello World'

    messy = "  too   many    spaces   "
    print(f"normalize: '{normalize_whitespace(messy)}'")
    # normalize: 'too many spaces'

# ── textkit/analyzers.py ──
"""Text analysis utilities."""
from collections import Counter

def word_count(text):
    """Count the number of words in text."""
    return len(text.split())

def char_frequency(text):
    """Return a Counter of character frequencies (lowercase, no spaces)."""
    return Counter(text.lower().replace(" ", ""))

if __name__ == "__main__":
    sample = "Hello World Hello Python"
    print(f"Word count: {word_count(sample)}")
    # Word count: 4

    freq = char_frequency(sample)
    print(f"Top 5 chars: {freq.most_common(5)}")
    # Top 5 chars: [('l', 5), ('o', 3), ('h', 2), ('e', 1), ...]

# ── textkit/formatters.py ──
"""Text formatting utilities."""

def title_case(text):
    """Convert text to title case, handling common small words."""
    small_words = {"a", "an", "the", "and", "but", "or", "in", "on", "at", "to", "of"}
    words = text.lower().split()
    result = []
    for i, word in enumerate(words):
        if i == 0 or word not in small_words:
            result.append(word.capitalize())
        else:
            result.append(word)
    return " ".join(result)

def truncate(text, max_len=50):
    """Truncate text to max_len characters, adding ellipsis if needed."""
    if len(text) <= max_len:
        return text
    return text[:max_len - 3].rstrip() + "..."

if __name__ == "__main__":
    title = "the quick brown fox jumps over the lazy dog"
    print(f"Title case: '{title_case(title)}'")
    # Title case: 'The Quick Brown Fox Jumps Over the Lazy Dog'

    long_text = "This is a very long sentence that needs to be truncated"
    print(f"Truncated: '{truncate(long_text, 30)}'")
    # Truncated: 'This is a very long senten...'

# ── main.py ──
"""Demo of the textkit package."""
from textkit import strip_html, word_count, title_case, truncate

# Clean HTML content
html = "<h1>Breaking News</h1><p>Python 4.0 announced!</p>"
clean = strip_html(html)
print(f"Cleaned: {clean}")
# Cleaned: Breaking News Python 4.0 announced!

# Analyze
print(f"Words: {word_count(clean)}")
# Words: 5

# Format
print(f"Title: {title_case(clean)}")
# Title: Breaking News Python 4.0 Announced!

print(f"Short: {truncate(clean, 20)}")
# Short: Breaking News Py...

📏 Best Practices

✅ Do's

• Always use a virtual environment — even for small scripts. It takes 5 seconds and prevents future headaches
• Name your venv venv or .venv — it's a convention that tools and IDEs recognize automatically
• Keep requirements.txt updated — run pip freeze > requirements.txt whenever you add or remove packages
• Use if __name__ == "__main__" — in every module that has runnable code, even if "nobody will import it." You never know.
• Include an empty __init__.py in every package directory — it's explicit and ensures correct import behavior
• Add venv/ and __pycache__/ to .gitignore — these should never be in version control
• Use python -m pip instead of bare pip — guarantees you're using the right Python's pip

❌ Don'ts

• Don't install packages globally with sudo pip install — this pollutes the system Python and can break OS tools
• Don't commit your venv/ directory — it's large, platform-specific, and unnecessary if you have requirements.txt
• Don't name files after standard library modules — random.py, json.py, email.py will shadow the built-ins
• Don't use from module import * in production — it pollutes the namespace and makes it hard to track where names come from
• Don't put .env files with secrets in Git — add .env to .gitignore and use .env.example as a template

💡 Pro Tips

• Use python -m venv --upgrade-deps venv to create a venv with the latest pip and setuptools pre-installed
• VS Code and PyCharm auto-detect venv/ and .venv/ directories and activate them for you
• Use pip install -e . (editable install) during development so you can import your package without reinstalling after every change
• The pipdeptree package visualizes your dependency tree: pip install pipdeptree && pipdeptree
• Consider uv as a drop-in pip replacement — same commands, 10-100× faster installation

📝 Summary

📚 Additional Resources

• Python Docs — venv
• pip User Guide
• Python Docs — Modules
• Python Packaging User Guide
• PEP 621 — pyproject.toml Metadata

🚀 What's Next?

In the next lesson, we'll put your environment skills to work with Working with APIs — using the requests library to interact with REST APIs, parsing JSON responses, handling authentication, and building a data dashboard.