The final thorn in my side when building the TwitterSpitter was something totally out of my hands. Twitter limits how many times you can request data from its services to 150 times per hour. This on its own is manageable but I had a problem where my rate limit was mysteriously diminishing at a very fast rate.
Watch the Video
It’s one thing to have account for the Twitter rate limit, its another to figure out how it is mysteriously going down. I literally spent days scratching my head over how it could be happening. My app was not public so I knew that no one was using it, yet Twitter kept saying that my requests were exceeding the limit every 45 minutes!
The Hunt
In order to track down the issue I de-activated every plugin that I use with WordPress that had any ties to social media services. I thought the Disqus comment system was to blame. I contacted them via Twitter and they were very helpful and quick to respond. All Disqus functionality is embedded in an iFrame so Twitter would not be seeing my IP address.
The Problem: Shared Web Hosting
After crossing everything WordPress related off my list, I found an article or forum post that mentioned that some web hosts may have multiple sites running off the same IP address. I researched my hostgator plan and lo and behold that’s what was happening. For all I know there could be 500 sites with my IP address. If any of those sites has any sort of Twitter widget installed, every time those widgets load, my rate limit was being effected. Literally I was losing 150 requests every 45 minutes.
The Solution
I could upgrade my plan and for an additional $2 per month I could have a dedicated IP address. Not a bad deal. Being that this app was largely an experiment, I didn’t think that was necessary. Fortunately, I have another totally separate hosting account for my old site www.doyouhaveapen.com (hasn’t been updated since 2004). It has a dedicated IP and I saw that my rate limit was accurately reflecting MY usage of the Twitter services.
Rate Limit Fallback
Since my TwitterSpitter is largely experimental and there is no real audience or purpose for it, I am fine with just offering an error message should the rate limit be exceeded. Its not like I have hordes of paying customers to keep happy. When ever I make an xml request of Twitter I check to see if the xml file contains an error node and then I check to see what message it contains.
if (xmlError == "Not found")
{
error_mc.message_txt.text = "That user was not found, please try again.";
}
else if (xmlError == "Not authorized")
error_mc.message_txt.text = "User account is protected. No dice.";
}
else if (xmlError.indexOf("exceeded") > -1)
{
error_mc.message_txt.text = "Twitter's annoying rate limit has been exceeded. Please try back in an hour 
";
}else
{
//generic response for other unanticipated errors
error_mc.message_txt.text = "Something went wrong, please check the wires";
}
launchError();
Screenshot of what happens when the rate limit error is encountered
screenshot of error
How do you test rate limit errors?
This was a real hassle, as once I realized that the rate limit was exceeded, it would always reset itself within a few minutes. Being that I couldn’t force the rate limit to go over without making hundreds of requests, I grabbed the xml of the error message and saved it locally. I would then tell LoaderMax to load this rate_error.xml file so that I could test if my app was responding properly.
The rate limit exceeded error looks like this
//rate_error.xml /1/statuses/followers/snorkltv.xml Rate limit exceeded. Clients may not make more than 150 requests per hour.
In fact, I made xml files of all the errors so I could load them and respond to them.
What you should really do
Technically your app should first check to see if the whether or not the rate limit has been exceeded via
http://api.twitter.com/1/account/rate_limit_status.xml
*note: checking your rate limit status does not count against your remaining hits, or so I’m told.
If the rate limit has been exceeded you should have a cached version of the data you want to show saved somewhere on your server. To avoid the rate limit altogether, you could have a script on your server that gathers the latest Twitter data you need once an hour. Your swf would then just load the most recent data off YOUR server.
For most Twitter widgets that simply show the latest 20 tweets of a user, a user’s Twitter timeline, or a list of followers, this would be fairly easy to update a cached version of this data if you have any backend knowledge. Totally missing from this discussion is the fact that Twitter also has a STREAMING api that allows info to constantly feed into your app with no rate limit. Streaming requires authentication which is way beyond what I’ve been able to tinker with. Incase you haven’t been paying attention, the rinky-dink public Twitter API requests really put me through the wringer;)
For the TwitterSpitter in its current form, it allows you to request data on any user whenever you want. There is no way to have this info cached. The TwitterSpitter is more of a proof-of-concept than an actual app that would hold up against any sort of real world usage.
Conclusion
Although the experience of building the TwitterSpitter forced me to deal with a few hefty unexpected challenges, I’m so glad I got to get it working in a way that I find to be acceptable. I was really impressed with how much you can do without knowing a whole lot. I’ve dealt with crossdomain issues plenty of times before, I really should not have waited til the end to test. I guess I was hoping Twitter would just have a friendly crossdomain.xml that would let everybody in. I’ve used php-proxies before when loading RSS feeds into Flash, I guess I just let my excitement get the best of me.
I was initially hoping I could twist the TwitterSpitter (with some extended functionality / eye-candy) into something people might pay $5 bucks for on www.activeden.net. I had ideas of loading each user’s latest Tweet when you clicked on their image and some other stuff. With all the server dependencies and possible rate limit issues, it seems like the potential support issues could be a real hassle.
I wouldn’t want to sell something that MIGHT work on your server, would require you to contact support and turn on proxy capabilities, or would require you to make sure that your host isn’t tying 100s of accounts to one IP address.
the good news
I learned a ton about what I need to research in order to get something like this totally bullet-proof. I also had a blast working with LoaderMax and may very well use the basic concepts (loading 100s of images based on xml data) into a tutorial in the very near future. I’ve already gotten a request to talk about how I handled the animation of everything spinning around randomly and I will definitely be sharing that.
I also got to document all the hassles I encountered so that anyone starting a little widget like this will know what things to look out for. This little series was probably the biggest undertaking of snorkl.tv. I’m confident that if people have the patience to read and watch it all, they will learn one or two things that will save them time in the future.
Other Articles In This Series
-Intro to the TwitterSpitter and common errors when loading XML data from Twitter
-Detecting illegal image names prior to loading
-Loading assets from other domains via a php proxy
Stay tuned, I have a bunch of tutorials in the works.
