How to interrupt gracefully in Python

I had a usecase where I was running a loop with each loop doing complex work. I wanted to run the code but also parallel improve it. The code was time consumingly gathering and processing data.

The improvements were future proofing it.

I wanted to kill the long running process in the terminal but not loose any information. So I learnt using ChatGPT how to do this in python.

The idea is:

1. Import the signal module.
2. Define a handler function for SIGINT that sets a global flag, indicating an interrupt signal was received.
3. Check this flag at a suitable point in your loop to decide whether to continue or break out of the loop.

import json
import logging
import os
import random
import re
import time
from pathlib import Path
from dotenv import find_dotenv, load_dotenv
from tqdm import tqdm
import signal  # Import signal module

# Configure logging
logging.basicConfig(
  level=logging.INFO,
  format='%(asctime)s - %(levelname)s - %(message)s',
  handlers=[
    logging.FileHandler('scraper.log'),
    logging.StreamHandler()
  ]
)


interrupted = False  # Global flag to indicate if interrupt signal was received

def signal_handler(signum, frame):
    """Handle interrupt signal."""
    global interrupted
    interrupted = True
    logging.info('Interrupt signal received. Will exit after this main() loop.')

signal.signal(signal.SIGINT, signal_handler)  # Register the signal handler


def main():
    # Your main function implementation remains the same
    pass

if __name__ == '__main__':
    for i in tqdm(range(50)):
        if interrupted:
            logging.info('Exiting loop due to interrupt signal.')
            break  # Exit the loop if interrupted flag is set
        time.sleep(random.randint(1, 5))
        main()

This is a neat pattern that is useful for my data engineering work in general where things are happening in iterations and I want to interrupt but not loose the work done. Also, partial failures are painful to deal with so thats a nice thing too.