Stata and the Twitter API (Part I)
Why Care Block:
This is it – the culmination of all of our hard work and it’s broken up into two posts. The first (this one) shows us how to get Twitter data, and the second shows us how to post things on Twitter using Stata and our API library.
This post is brought to you thanks to an openness and willingness to share information. In today’s day and age, it’s ridiculous to try and hide information or impede the transfer of knowledge to others. I recently experienced first-hand the negative result openness can have on those who wish to obfuscate information. I guess these certain individuals are incapable of adapting, sharing, and participating in the global community – perhaps it’s not their fault if they feel threatened, but this does not excuse what I believe to be an original sin when it comes to technology. Sure it’s difficult to produce new things and sometimes even more difficult to scrap projects you’re emotionally attached to, but to abandon creativity and sequester change is the antithesis of this post. I’m sure many of you have dealt with an issue similar to this in your line of work, please don’t give up the good fight. If this sounds familiar, well this post is dedicated to you.
Just like the last Fitbit API post, all corresponding variables are color-coded for your convenience. The first thing we need to do is set up an application with the Twitter API. Once you enter all the necessary information, go to the “Keys and Access Tokens” tag to get your Consumer Key and Consumer Secret. FYI, all information provided below is fake, sorry to disappoint.
Below the Application Settings section is the “Your Access Token” section which contains the Access Token and Access Token Secret which we will use to get our basic tweet data and search for other interesting things.
Just like our Fitbit API, we need to first get an Authorization Token by sending a cURL request that takes the Base64 encoded consumer key and client secret. First let’s set up our variables:
local cons_key "ZAPp9dO37PnzCofN2Nm8n8kye" local cons_sec "kfbtARFpBIdb515iaS48kYZjWhLoIdbEiAINDVX0c3W3e0fgWe" local accs_key "1234567890-YFklDWGuvSIGLYPMnAfOZgLgLsMXjKIHqaIr1F5" local accs_sec "LYvHWfTS6LXjtDPVXchs6dXUG52l41j4HmicYwjr8aStw"
Note: we will not be using the access key or secret until the next post
And create our encoded consumer information string (recall encode64 is available through this post):
encode64 "`cons_key':`cons_sec'" local client64 = r(base64)
Now we simply send our request to Twitter:
!curl -k -i -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Basic `client64'" --request POST --data "grant_type=client_credentials" https://api.twitter.com/oauth2/token --output "token.json"
Be sure to parse the access token from our outputted token.json file (perhaps this would be a good place to enter error handling methods), I stored the value in the global macro $token. Finally, send one more request that contains the query you’re looking for:
!curl -k -i -H "Authorization: Bearer $token" --request GET "https://api.twitter.com/1.1/statuses/user_timeline.json?count=100&screen_name=_belenchavez" --output "output.json"
In this case, we were able to pull the top 100 things from @_belenchavez’s timeline using the statuses API. For more information about what you can do with the application only authentication, check our Twitter’s documentation here. Otherwise, take a look at this frequency cloud of my commonly Tweeted/Re-Tweeted hashtags of 2016 thanks to wordclouds.com. It even looks somewhat like the Twitter logo!
4/2/2016 11:26:40 pm
The verification process does have a sensible function. Twitter brims with bogus or parody accounts. So when customers are sifting via a summary of likely usernames, it can help to get alerts to aid find the precise man or woman they want to abide by.
7/31/2016 08:40:22 am
This is a very interesting post here on what you did with the Twitter API. I had no idea about this, thank you.
12/3/2016 10:07:20 am
This is a fantastic website and I can not recommend you guys enough.
12/26/2016 11:10:17 am
Thank you William for this very interesting post! I have been collecting data from twitter and youtube using python, but I would rather do it directly from Stata.
1/25/2018 02:27:52 am
I really enjoyed reading this post, big fan. Keep up the good work andplease tell me when can you publish more articles or where can I read more on the subject?
10/9/2022 01:47:25 pm
Very thoughtful blogg
Leave a Reply.
Will Matsuoka is the creator of W=M/Stata - he likes creativity and simplicity, taking pictures of food, competition, and anything that can be analyzed.