The NYC School Survey: Examining Over- and Under-Performing Schools

Project Description

 

This project was completed for Advanced Projects in Visualizations with instructor Can Sucuoglu. The purpose of this project was to examine the thoughts, attitudes, and opinions of students and teachers in schools from NYC's five boroughs with the research question:

"How do the thoughts, attitudes, and behaviors of students and teachers in under-performing schools compare to students and teachers in over-performing schools?"

This project particularly focused on examining differences in middle and high schools. To answer this question, I used a dataset which I created by using a Python script to scrape Greatschools.org in order to create a file with information on schools in NYC in tandem with data from the NYC school survey from 2015-2019. The overall result of this project was a static webpage which contained a number of visualizations created using Tableau Public and in-depth analysis of the findings. This projects focused primarily on understanding students and teachers' behaviors in both under- and over- performing schools through some exploratory analyses. The results of this project found that there were not any major differences in the thoughts, attitudes, and behaviors of students and teachers in both under- and over- performing schools.

​

Method

​

To begin this project, I first used a Python script to scrape data from Greatschools.org in order to create a dataset with information on the schools. Some of this information included name, district, school type, overall rating, enrollment data, etc. This script was initially written by my courses instructor for a project that he previously worked on and I need to make some adjustments in order to scrape all of the data that I needed. The next step of this project was clean the dataset up a bit. This done using  Microsoft Excel as most of the cleaning was to simply delete duplicate values and adding and defining grade bands for the schools. After cleaning, I defined which schools were under- and over-performing using Greatschools 10 point rating scale. Any schools that were rated a 1 were under-performing and 10 were over-performing. Greatschools uses their own grading rubric to determine the overall score of the school, based on a number of factors. The final result w as 25 over-performing schools and 25 under-performing schools.

Then I downloaded the survey results from the NYC department of educations websites for both students and teachers responses from 2015-2019. The surveys were slightly different from year to year in terms of question phrasing and placement, so I manually matched up each question from each survey, year over year on both student and teacher surveys. Then I cleaned and created datasets with the survey responses for the appropriate schools using Microsoft Excel and OpenRefine for both cleaning and manipulation. Most of this data manipulation took place using Microsoft Excel. Finally, I created several visualizations using Tableau Public and conducted an in-depth analysis on the survey findings. I even conducted a deeper dive on a specific school district that had both under- and over-performing schools (NYC school district 14) to attempt to control for extraneous variables. My visualizations and analysis were then placed on a wix hosted site.

​

My Role

​

I was responsible for all aspects of this project.

​

Learning Outcome Achieved 

​

Research - Students can develop complex questions surrounding data and select and apply appropriate methods to answer them.

​

Rationale

​

Research - This project demonstrates my ability to ask thoughtful research questions as well as selecting appropriate methods to conduct research and to answer my questions. My research question evolved from a very broad question, which asked "how has education in the US changed in the past few years?" to the question it is now. To arrive at this question, I began with secondary research to survey any existing works and found several articles on different trends over the past 20 years. As I searched, I found that the scope of my project was much larger than I could handle during the course and I constantly had to refine my question until I was able to develop a clear and focused question. This was after I stumbled upon the NYC school survey dataset and I was interested in examining the responses between student groups. I needed to supplement my data by gathering information on schools, which I was able to do using Python. By using Microsoft Excel and Tableau, I was able to transform and visualize the data into a medium that was easily digestible and useful for analysis.