Stata and the Twitter API (Part II)

OAuth

Why Care Block:
You can Tweet things in an automatic fashion directly from Stata

Oh man, things are about to get real juicy – this is the final reveal of the Twitter API and I couldn’t think of a better time to do this.  After all, we’ve made a complete mess out of Stata with our unruly bitwise operators, unending computer security algorithms, and un-um… unfortunate time-based C-plugins.  Well, my Stata friends, this is how to do anything with Stata and the Twitter API.

Let’s post!  Seriously, let’s tweet about things using Stata.  This method can be used not only to tweet, but also to send direct messages, retweet things, and like (favorite) others’ tweets.  If you’re like me, you don’t really use Twitter that well and I’m sure my 40 followers (I rounded up) would agree.  But wouldn’t it be great to find a way to reach thousands of Twitter followers based off of hashtags or messages or even emoji?  If the answer is no, fine, that’s my best attempt at convincing you but still feel free to read on.

Let’s bring together everything we need to send a Tweet out on the internet:

• Stata: Check
• cURL: Check
• Our Library: Check

​And what does our library now consist of? (FYI: there’s some new stuff here)

Base64 Encoding

​Easy way to turn readable stuff into unreadable stuff for humans but it’s not so bad for computers.  This takes all alphanumeric characters plus the plus and the “/” (which equals 64 characters to choose from), converts 8-bit chunks into 6-bit chunks which end up receiving the specific character.  Sometimes equal signs are added, but you already know that.

See post on Base64 Encoding

​Bitwise Operators and Security Methods

We’ve got bits dressed to the nines: shifting operators, rotating operators, logical operators, overflow methods, and all kinds of methods to translate from base 2 to base 16 to base 10.
​
These feed into our HMAC-SHA1 algorithm, which was awful to develop… this seriously has to go in the “lots of effort, little reward” category. 

​See post on Bitwise Operators and HMAC-SHA1

Stata Plugins

​We needed time - I mean we all do – in seconds since 1970 based off of Greenwich Time.  We could have just accounted for our respective time zones (unless you’re located in GMT+0 of course), but why not use this as a great learning opportunity?

See post on Plugins

​Nonce

​Once?  None?  Scone?  What’s a nonce?  Simply put, it’s a random 32 character word containing all alphanumeric symbols.  Twitter’s example shows this being Base64 encoded, but all it has to be is a random string.  vqib2ve89dOnTusESAS26Ceu9TcUES2i – see?  Easy, just make sure you change the characters each time.

Percent Encoding

What%27s%20percent%20encoding%3F
​
Based on the UTF-8 URL Encoding standards, we need to replace certain characters that would otherwise cause complications in a URL.  Don’t worry though, we’ve got a function for that.  Just know that valid URL characters are all letters and numbers as well as the dash, the period, the underscore, and the tilde. 

twitter-programs-lib.do
File Size:	10 kb
File Type:	do

Download File

---

---

​Send Out the Tweets

​Twitter has some fabulous documentation for their API so it’s fairly easy to find the method that you want to do.  These processes can be generalized and used across many of these methods so we are by no means limited to just tweeting.  For tweets (hereon referred to as status updates to be consistent with Twitter’s documentation), we use the base or “resource” URL:

https://api.twitter.com/1.1/statuses/update.json

​​Remember, status updates can only be 140 characters and Twitter will actually shorten URLs for you, but still keep in mind your character limits!

All statuses must be percent encoded, so let’s go ahead and create our status local; we’ll also store our base URL while we’re at it.

local b_url "https://api.twitter.com/1.1/statuses/update.json"
local status = "Sent out this tweet using #Stata and the #Twitter #API – http://www.wmatsuoka.com/stata/stata-and-the-twitter-api-part-ii tells you how!" 
mata: st_local("status", percentencode("`status'"))

​We then take all of our application information: consumer key, consumer secret, access token, and access secret and store these as locals as well.

local cons_key "ZAPp9dO37PnzCofN2Nm8n8kye"
local cons_sec "kfbtARFpBIdb515iaS48kYZjWhLoIdbEiAINDVX0c3W3e0fgWe"
local accs_key "1234567890-YFklDWGuvSIGLYPMnAfOZgLgLsMXjKIHqaIr1F5"
local accs_sec "LYvHWfTS6LXjtDPVXchs6dXUG52l41j4HmicYwjr8aStw"

For this section, we will be creating all of our necessary authentication variables.  The first step to creating our secret signing key is to make a string that contains our consumer secret key, concatenated with an ampersand, concatenated with our access secret key.  We’ll also include all the necessary locals we talked about earlier.

local s_key    = "`cons_sec'&`accs_sec'"
mata: st_local("nonce", gen_nonce_32())
local sig_meth = "HMAC-SHA1"
// If you don't make a plugin, just make sure you do seconds since 1970
// probably using c(current_date) and c(current_time) - [your time zone]
plugin call st_utm
local ts = (clock(substr("`utm_time'", 4, .), "MDhmsY") - clock("1970", "Y"))/1000

​Now it’s time to create our signature.  We start by percent encoding our base URL.  For our signature string, we include the following categories:

• Oauth_consumer_key - `cons_key’
• Oauth_nonce - `nonce’
• Oauth_signature_method – “HMAC-SHA1”
• Oauth_timestamp - `ts’
• Oauth_token - `accs_token’
• Oauth_version – “1.0”
• Status – Whatever we want our status to be (percent encoded of course)

Note how they’re in alphabetical order by category – it is very important that you organize your string in this order.  This is how it looks:

local sig = "oauth_consumer_key=`cons_key'&oauth_nonce=`nonce'&oauth_signature_method=`sig_meth'&oauth_timestamp=`ts'&oauth_token=`accs_tok'&oauth_version=1.0&status=`status'"

Then, percent encode the signature string and base URL:

mata: st_local("pe", percentencode("`b_url'"))
mata: st_local("pe_sig", percentencode("`sig'"))

This next step is why we spent an ungodly amount of time on bits and hashes!  We need to transform our signature string into a Base64 encoding of the HMAC-SHA1 hash result with our message being the percent-encoded signature base hashed with our secret key.  Sorry if that's a mouthful, but we can see what that translates to below.

local sig_base = "POST&`pe'&"
mata: x=sha1toascii(hmac_sha1("`s_key'", "`sig_base'`pe_sig'"))
mata: st_local("sig", percentencode(encode64(x)))

Finally, we’re almost done with Twitter and can move on to other more important things!  First let’s make sure to post our Tweet because, after all, that's what we're here for.

!curl -k --request "POST" "`b_url'" --data "status=`status'" --header "Authorization: OAuth oauth_consumer_key="`cons_key'", oauth_nonce="`nonce'", oauth_signature="`sig'", oauth_signature_method="`sig_meth'", oauth_timestamp="`ts'", oauth_token="`accs_tok'", oauth_version="1.0"" --verbose

Now we can marvel at how much time we spent learning about something 99% of us don’t really care about.  This is for you, 1% - this is all for you.

Sent out this tweet using #Stata and the #Twitter #API – https://t.co/iX3kBd2VFu tells you how!

— William Matsuoka (@WilliamMatsuoka) March 4, 2016

PS: Want to add emoji?  Go to this site, find the URL Escape Code sections, and add the result to your status string!

Ex: %F0%9F%93%88 = Chart with Upward Trend ​