IS 930 Web Scraping for Researchers

Content

In today’s data-driven world, the ability to obtain web data is a game-changer for empirical research. Top-tier research papers in marketing, management, accounting, finance, and information systems increasingly rely on data obtained from scraping websites or querying APIs. Examples of such data include customer reviews from Amazon, search trends on Google, Instagram posts, job postings, investment disclosures, and much more. The skill to obtain novel, rich, and unique web data has become critical for business researchers. 

This hands-on course equips doctoral students and postdocs in business with the essential skills to collect, extract, and process web data for research purposes. You will follow a structured, step-by-step approach to:

• Identify requirements for effective data extraction
• Automate data collection from websites or APIs
• Parse and clean data
• Navigate ethical and legal considerations

Through practical exercises in small groups, you will develop a fully functional web scraper using the Python programming language, which you can easily adapt for your own research projects later on.

You do not need to be familiar with Python to attend the course. If you have not worked with Python yet, we provide setup instructions before the course starts.

Learning Outcomes

By the end of the course, students ..:

• ..can evaluate the costs and benefits of automating data extraction from websites and APIs,
• ..can implement web scraping techniques in Python to collect, parse, and clean data from websites and APIs,
• ..can address common technical challenges in web scraping, including dynamic content and scalability,
• ..can explain ethical and methodological considerations related to web scraping.

Necessary prerequisites:

–

Recommended prerequisites:

–