Project: Final

Python

Overview

Each student will choose a dataset related to a topic they are interested in that produces a research paper employing the data analytics methods taught in this course. You should explore the data and make a clear argument that is evidence-based. This is a written assignment that shows both your problem-solving logic and data analytics methods using Python. That means it should use Markdown to produce a typical essay with an introduction (including a thesis) and conclusion as well as body paragraphs. Much of this class is about data transformation, so make sure to describe how you had to transform your data to make the statistical analysis possible. Use code cells to include thorough Python code to provide the statistical evidence that proves your argument. The minimum criteria are provided below. I have attached links to various options below. You will turn in a Jupyter notebook (.ipynb) for each submission.

There will be multiple checkpoints when you will turn in a portion of the final project. Each checkpoint will receive its own grade. You are also permitted to use any work in the check points in the final project.

Note

All assignments associated with the final project must be completed in order to receive credit for the final project.

There will be four (4) such checkpoints:

  1. Dataset & Question (end Week 3)
    1. Choose a dataset.
    2. and a question
  2. Calculate Basic Statistics (Week 7)
  3. Variable analysis & visualizations (week 13)
  4. Final due Date (Week 16)
  5. Process Paper (Week 16)
Note

The Final Project is completed through connection with the GitHub classroom. Once you have created a GitHub account (with your Queens email), you will be able to join the classroom and access the assignments for the Final Project.

See the instructions for connecting to the GitHub Classroom

Organization

The final project should be organized as a Jupyter notebook (.ipynb) that acts as a data analytics report to be turned into a manager or client. Think of it as an essay format. It should include the following:

  • Markdown explaining your problem, solution, and the process you went through to get the answer.
    • This includes an introduction and a conclusion.
  • It should have an introduction and a conclusion.
  • There should be Markdown paragraphs describing the steps in the process.
    • What is the code doing?
      • If you are transforming the data. Why are you transforming it that way? To what end?
    • Why are you doing it?
  • It should indicate what explain the statistics.
    • Describe not only what the statistic actually means (the f surpasses the critical-f, so it is significant, for example) but what that means in the context of your data and problem.

Criteria

The final project must include the following to receive full credit

  • At least 3 visualizations of different types aligned to each test.
  • At least 1 correlation calculation
  • At least 1 t-test
  • At least 1 ANOVA test
  • A written explanation of the results of the t-test and ANOVA test
    • The explanation should include the null hypothesis, the alternative hypothesis, the p-value, and the conclusion.
    • Explain why you chose the variable you did for the t-test and ANOVA test.
Note

See the rubric here: Final Project Rubric

Possible Datasets By Area

You are not limited to these datasets. You can choose any dataset you would like. These are just some suggestions to get you started. Also, you have to look around and click around. The sites do not all look the same. Some are more user-friendly than others. Some have more data than others. You might need to combine data or clean data. Data analysis is about solving problems. Some problems will be related to aligning the data correctly.

Good places to search for datasets are:

Business and Finance

  • Yahoo Finance API
    • Yahoo Finance has an expansive API that allows you to pull in stock data with extensive amounts of information. This is actually a Python library that allows you to interact the API more easily. It is available by default in Colab.
  • World Bank Open Data
    • The World Bank has a variety of data on countries around the world. You can search for data by country, topic, and more.
    • The World Bank data has a significant amount of data.

Sports

  • NFL Play-by-Play Data
    • This package allows you to pull in NFL play-by-play data. This data is very detailed and can be used to answer a variety of questions about the NFL. This is also a Python library that allows you to interact with the API more easily. It is not available by default in Colab. You can add it by running !pip install nfl_data_py in a code cell then import it.
  • NFL Data
    • This dataset contains a variety of information about NFL games, including scores, betting lines, and more.
  • World Cup 2018 Event Data
    • This data is available on my GitHub account. It was collected by Luca Pappalardo and Emanuele Massucco and is available on the figshare platform. There is also a GitHub repo explaining how to work with the data (here)[https://github.com/Friends-of-Tracking-Data-FoTD/mapping-match-events-in-Python]. I have cleaned the data to make it easier to work with. It has every event in every game of the world cup and where on the field it took place: passes, shots, challenges, etc. It comes in JSON format. You can find the raw data here. You will want to use the url for the raw data when you request the data in your Python script. This is super cool data of a similar granular quality as the NFL play-by-play data.

Climate and Environment

  • NOAAA Climate Data Online
    • The National Oceanic and Atmospheric Administration has a variety of climate data that you can use for your project. You can search for data by location, date, and more.

Search for Other Data

  • Kaggle Datasets
    • Kaggle has a variety of datasets that you can use for your project. You can search for datasets by topic or popularity.

GitHub Classroom

This assignment uses GitHub Classroom. GitHub Classroom is a tool that helps instructors manage coding assignments using GitHub repositories. When you accept the assignment, GitHub Classroom will create a private repository for you where you can work on your code. To complete this assignment you will have to accept the assignment, clone the repository to your local machine, complete the assignment, and then commit and push your changes back to GitHub.

To accept the assignment, go to the assignment page on Canvas and click on the link to create the GitHub repository.

Note

Part of this assignment is not just completing the coding tasks, but also learning how to use GitHub and Git for version control. Make sure to follow the instructions carefully and reach out to me if you encounter any issues.

Understand, problems with Git/GitHub are not an excuse for late work. Plan ahead to ensure you have time to resolve any issues that may arise. In industry, version control is an essential skill, and being proficient with these tools is crucial for collaboration and project management. Moreover, technology issues (including those with GitHub) will never be accepted as an excuse for missing deadlines. This is a programming class. The technology is part of the assignment.

If technology difficulties arise, I’m happy to help, but before deadlines (not after).

Cloning the Repository

Once you have accepted the assignment, a new repository titled - will be created in your GitHub account. You can log in to GitHub to view the repository. You will need to clone this repository to your local machine to work on the assignment. Follow these steps to clone the repository:

  1. Open VS Code
  2. Press Ctrl+Shift+P (or Cmd+Shift+P on Mac) to open the command palette.
  3. Type Git: Clone and select it.
  4. Select Clone from GitHub...
  5. You might be prompted to sign in to GitHub if you haven’t already.
  6. After signing in to GitHub, you will need to follow steps 1-4 again to clone the repository.
  7. You should receive a list of your repositories. Select the repository for this assignment.
    1. If you do not see your repository, you can copy the URL from GitHub and paste it into the prompt in VS Code.
  8. Choose a local directory where you want to clone the repository.
    1. Choose the folder location carefully. File management is incredibly important in programming, and it is best to keep all your assignments for this class in one folder.
    2. Make sure NOT to clone the repository into another repository.
  9. Click Select Repository Location to complete the cloning process.

Committing and Pushing Changes

In order to submit your assignment, you will need to commit any changes you make to your local repository and push them to GitHub. Committing saves your changes to the local repository, while pushing uploads those changes to the remote repository on GitHub. In other words, just saving your files is not enough. Saving updates the files on your computer, but when you commit those saves, you are creating a snapshot of your project at that point in time. Think of a commit like a save point in a video game that you can always return to. Remember, you can commit multiple changes to multiple files at once You should commit your changes to the repository often, especially after completing significant portions of the assignment. To do so, follow these directions.

  1. Click the Source Control icon in the Activity Bar on the side of VS Code (it looks like a branch with a dot at the end). You should see a list of files that have been changed.
  2. Stage changes to be committed. Not all files need to be in every commit. You can stage individual files by hovering over the file name and clicking the plus (+) icon that appears, or you can stage all changes at once by clicking the plus (+) icon next to the “Changes” header.
  3. Enter a commit message in the text box at the top of the Source Control panel that describes the changes you made. A good commit message is concise yet descriptive, such as “Completed function to calculate average” or “Fixed bug in data processing loop.”
  4. Press the blue button below the commit message box that looks like a checkmark to commit your changes.
  5. After committing, these changes are still only saved locally. The blue button below the commit message input box now states “Sync Changes” with an up and down arrow icon. Click this button to push your committed changes to the remote repository on GitHub. You may be prompted to sign in to GitHub again if you haven’t done so recently.
  6. Check the GitHub repository online to ensure your changes have been pushed successfully. If you cannot see your changes online, try refreshing the page. If you still cannot see your changes, the process was not completed successfully, and you should try pushing again.