How to automate data scraping from websites using Python?

Step 1: Install Required Libraries
To start with web scraping in Python, you need to install two very popular libraries used for web scraping i.e., BeautifulSoup and Requests. You can install them using pip:

pip install requests beautifulsoup4

Step 2: Import Required Libraries
Once you have installed the required libraries, you need to import them into your Python environment.

import requests
from bs4 import BeautifulSoup

Step 3: Make a GET request
You need to make a GET request to the URL of the webpage you want to access. The server responds to the request by returning the HTML content of the webpage.

URL = "http://www.example.com"
r = requests.get(URL)

Step 4: Parse HTML content
Next, you need to parse this HTML content and for that, you will be using BeautifulSoup.

soup = BeautifulSoup(r.content, 'html5lib') 

Step 5: Searching and navigating through the parse tree
Now you can search and navigate through the parse tree that you created, i.e. BeautifulSoup object. You can search for a tag by using the tag as a string (like "a" for hyperlinks, "table" for tables, "div" for divisions, "b" for bold etc) as an argument.

quotes=[]  # a list to store quotes
  
table = soup.find('div', attrs = {'id':'container'}) 
  
for row in table.findAll('div', 
                         attrs = {'class':'quote'}): 
    quote = {} 
    quote['theme'] = row.h5.text 
    quote['url'] = row.a['href'] 
    quote['img'] = row.img['src'] 
    quote['lines'] = row.h6.text 
    quote['author'] = row.p.text 
    quotes.append(quote) 

Step 6: Save the data
After extracting, you might want to store the data. You can store the data in a CSV file.

import csv
filename = 'quotes.csv'
with open(filename, 'w') as f:
    w = csv.DictWriter(f,['theme','url','img','lines','author'])
    w.writeheader()
    for quote in quotes:
        w.writerow(quote)

This is a basic guide to start web scraping with Python. Depending on the complexity of the website and the data, the process can be more complicated. You might need to deal with different structure of the website, javascript elements, login requirements and so on.