GEOG5790M: Programming for Spatial Analysts: Advanced Skills: Assessment 2 - Independent Project

During my MA in Social Research at the University of Leeds, I was assigned to do an independent project as part of the Programming for Spatial Analysts: Advanced Skills module. As the assessment was an open project, I decided work on a project that I'm currently interested in; Tweet scraping using Python.

This project contains two parts:

Part 1: Building a script to stream live-Tweets on a topic, and storing the data into a MongoDB collection. Then converting the data into a CSV file for analysis.

Part 2: Building a set of tools containing scripts to analyse the Twitter data. Scripts built allow the following:

The creation of a table to compare the number of people Tweeting from different time zones
The creation of a time-series area chart of the Tweets over the time period of streaming.
The mapping of Tweet's using Tweets that have geolocation enabled.
The creation of a pie chart to analyse what platforms and sources the Tweeter's used to send their Tweets, such as Twitter for Android, iPhone, or the Web Client.
The creation of charts that displays the frequency distribution of the top 25 most common words from Tweets in the entire database.

To demonstrate the use of the script, my project streamed live-Tweets on the topic of the Royal wedding of Price Harry and Meghan Markle on Saturday the 19th of May at 11:00 GMT. To view the code of each script, the output on the topic of the Royal wedding, and the requirements to run the scripts, please click this link to be taken to the corresponding GitHub repository.

Guide to the GitHub Repository Structure

The repository contains two key folders: "1. Stream-Convert-Tweets" and "2. Analyse-Tweets" which hold the scripts to run the code and the expected output.

Within "1. Stream-Convert-Tweets", you will find the following Python and CSV files:

TweetStream_RoyalWedding.py: contains code to connect to Twitter's API using Tweepy, scrapes Twitter for Tweets on the Royal Wedding topic (can be changed), and saves the data into a MongoDB collection.
MongoDB_to_CSV.py: contains code to read into a MongoDB collection, and then convert the data into a CSV file (which will be large if scraping Tweets over a long period).
RoyalWedding.csv: created using the MongoDB_to_CSV script, containing all of the data scraped.

Within "2. Analyse-Tweets", you will find the following Python and HTML files:

CreateTweetTimeSeries.py: contains code to create a time series chart of Tweets per minute.
CommonTimeZone.py: contains code to table the top 15 time zones of Tweeters.
MapTweets.py: contains code to map the geolocation of Tweets.
TwwetPlatformChart: contains code to create a pie chart of Tweet sources.
WordFrequencyAnalysis.py: contains code to create graphs of word frequency distrubtions of the top 25 most common words.

GEOG5790M: Programming for Spatial Analysts: Advanced Skills: Assessment 1

For the first assessment of the Advanced Programming module, practicals were provided for Data Processing and Data Analysis using Python and ArcGIS. We were taught the basics of how to use Arcpy, databases, XML, Numpy, Pandas, Bokeh, and NLTK.

To access my output from the practicals with more information, click this link to be taken to the corresponding GitHub repository.

Guide to the GitHub Repository Structure

The repository contains two key folders: "Data-Processing" and "Data-Analysis" which hold the practical data and outputs.
Within "Data-Processing", you will find the following folders:
- ArcPy-Addin-GUI: contains script .py files to run burglary tool created fro ArcMap. Also a Burglary-Addin.rar file containig an ESRI addin install of a created addin associated with the .py script. Also within this you will find an "albertsquare" folder containing shape files required for the script to run. Finally, a "Models.tbx" toolbox is provided holding the tools created for the practicals.
- Databases-SQLite: contains code to create an SQLite database, and a script to read the database.
- XML: contains XML which can be interacted with Python, and allows the user to validate, transform, and read XML files in various ways.
Within "Data Analysis", you will find the following folders:
- Pandas: this folder contains workshop practicals followed to learn how to use Pandas.
- Bokeh: this folder contains workshop practical output, which was used to learn how to use Bokeh and Geo Mapping.
- NLP: this folder provides a small project using Natural Language Processing. The first part reads into a poem and write all words which are place names into a .CSV file. The second part creates an interactive map of the places using Bokeh.

GEOG5990M: Programming for Spatial Analysts: Core Skills: Assessment 2

This is the second assessed project for my coding module using Python. This agent-based model has been coded without assistance or practicals to demonstrate how well I can code. The project's environment is a city, which simulates drunk people attempting to travel home.

Key Model Notes

The provided town plan (town.plan.txt) file is used to set the model environment and is read into the model row by row.
A new environment is created using the same layout as the town plan, this time with each cell being 0. This is used to create a density map.
Using the x and y coordinates, each agent is commanded to identify the location of the pub, and start off there when the model is initiated.
Agents are then allocated an ID number and house number (calculated as (ID#+1)*10).
Each agent is instructed to navigate the town in random directions until it has reached its home.
Each step taken is recorded on the new environment output, each step on a cell increases its value by one.
If an agent has not reached its home, it will continue to wander until it has, taking as many model iterations as required.
Once all agents have arrived home, the new environment file is saved and printed, producting a density map.
When an agent has reached home, it will call out "What a journey! WHERE'S MY BED!!", letting the user knowits arrival.
The intial map will reopen at the end of the model run, displaying each agent at their home.
Finally, a new window will open displaying the new environment map, coloured to show the density of each map cell.
The new environment will also be saved as a .csv, so the number of steps can be viewed.

You can access the final code and further information about the project by clicking this link which will take you to my GitHub repository.

GEOG5990M: Programming for Spatial Analysts: Core Skills: Assessment 1

Here you will find the link to my project work for an agent-based model that was built following practical sessions for the assignment named above. The module is part of my MA Social Research (Interdisciplinary) programme, which has taught me the Python language, allowing me to utilise this skill during my PhD at the University of Leeds.

Key Model Notes

The environment can be altered by the agents as they eat their way around.
The agents know that they are not alone in the model and can interact such as being able to share 'data' stores.
If an agent is in the way of another, it will move around the environment, and not on top of the other agent.
Each agent has a data store capacity of 1000, if it consumes any more data it will vomit, depositing all of its data stores.
The behaviours of each agent in their environment can be viewed as an animation. If a piece of data has been consumed by an agent, the environment patch will change colour and will have a decreased data store.

You can access the final code and further information about the project by clicking this link which will take you to my GitHub repository.