For Wil Wheaton
Here’s the un-bit.lyfied version to download your last 3,200 tweets in one shot on a machine with curl installed.
curl -O -u username:password "https://twitter.com/statuses/user_timeline.xml?count=100&page=[1-32]"
Replace username and password as necessary. If your password has a special character like an exclamation point or an ampersand, you’ll have to put a backslash (\) before it.
Twitter only allows you to retrieve your last 3,200 tweets, but I’ve got more archived over at TweetStats – just email or call with contact info from the sidebar.
Damon (@dacort)
Continue reading » · Written on: 07-16-09 · 37 Comments »
Hey Damon,
Thanks! This worked like a charm (and dislodged my memories of using curl, wget, and rsync as an added bonus.)
July 16th, 2009 at 6:28 pmWil,
Glad to hear it! It’s unfortunate that Twitter only currently allows a limited view of your timeline. I hope they’ll address it one day, but until then … this guy is keeping regular backups.
July 16th, 2009 at 10:36 pmHey, I think that only downloads 32*20=640 tweets, not 3,200, aren’t you missing a parameter that sets it to download 100 tweets at a time, not 20?
July 17th, 2009 at 11:35 amPerhaps you could indicate you want pages [1-160] and that will get 3,200 assuming 20 posts per page that twitter gives you?
July 17th, 2009 at 11:37 am[...] and our iPhone happily keeps track of every single one. Luckily, with some command line-fu, you can back up your Tweets if it’s that important to you, but in the case of your Facebook history, and many other sites, [...]
July 17th, 2009 at 11:43 amI tried this, and what you end up with if you try [1-160] is error message after 150 requests. You need to split it up between two different hours.
July 17th, 2009 at 11:46 amThis comment has no relation to Wil Wheaton, this post or the like. I couldn’t find any other contact point for you, so this is it, hoping you read it. It’s about your girlsmakingboysounds website! My girlfriend does a TERRIBLE transformers noise (you know the one…”chuk chek chegg chu chesssughhh”) and I’d love to have the world experience my laughter. I’m sure other gals have the fantastic inability to reproduce the same noise and this is my recommendation to have that added to the phone system. Let me know if that’s possible. Just drop me an email. Thanks!
July 17th, 2009 at 12:23 pm@K.M. – When you’re calling the API (as opposed to the website), Twitter retrieves 100 tweets per page by default. So the 32 pages will get you 3,200 updates – the maximum you are allowed to go back in your timeline.
The 150 requests is the API limit enforced by Twitter per hour.
July 17th, 2009 at 3:59 pm[...] DCortesi . blog » For Wil Wheaton (tags: twitter backup howto internet automation data curl) [...]
July 17th, 2009 at 8:01 pmHey Damon,
Are you getting 100 tweets from your curl command per page? Check the xml files that are produced by your command, in my test of your command, each of them contain only 20 entries, which I believe encapsulates a single entry.
July 18th, 2009 at 2:13 amK.M. – I stand corrected. Looks like the API changed or my memory is going.
I’ll update the command accordingly.
July 18th, 2009 at 6:27 am[...] Twitter backup using curl Here’s the un-bit.lyfied version to download your last 3,200 tweets in one shot on a machine with curl installed. (tags: Twitter backup curl) [...]
July 19th, 2009 at 4:04 pmJust a concern, not related to the post. I still use IE6 at work for some design stuffs. I see your homepage floats left with no left padding/margin. I thought to share with you.
July 23rd, 2009 at 1:39 pm@San Nayak
IE6 is dead and dying. Use PortableFirefox or Opera@USB if you can’t install a decent browser at work.
A lot of us no longer test in IE6 because it’s 8 years old and because we don’t have it on our machines.
Bug your IT guys at work, not web developers.
July 29th, 2009 at 11:23 am[...] line lovers can use this clever method to download their tweet XML via cURL. Alternately, web application Backup My Tweets does just that and lets you download your tweets in [...]
August 12th, 2009 at 8:32 am[...] line lovers can use this clever method to download their tweet XML via cURL. Alternately, web application Backup My Tweets does just that and lets you download your tweets in [...]
August 12th, 2009 at 8:40 am[...] line lovers can use this clever method to download their tweet XML via cURL. Alternately, web application Backup My Tweets does just that and lets you download your tweets in [...]
August 12th, 2009 at 8:56 am[...] line lovers can use this clever method to download their tweet XML via cURL. Alternately, web application Backup My Tweets does just that and lets you download your tweets in [...]
August 12th, 2009 at 10:23 am[...] line lovers can use this clever method to download their tweet XML via cURL. Alternately, web application Backup My Tweets does just that and [...]
August 12th, 2009 at 11:10 am[...] line lovers can use this clever method to download their tweet XML via cURL. Alternately, web application Backup My Tweets does just that and lets you download your tweets in [...]
August 12th, 2009 at 12:49 pm[...] os seus tweets — acredito eu, que poucos irão querer — existem vários métodos. O cURL, é uma linha de comando que usa um método eficaz para baixar os seus tweets por XML. Já o [...]
August 12th, 2009 at 5:12 pmI copied and pasted the command on my terminal but all i get is
>
Am I doing something wrong? OSX 10.5.8
August 13th, 2009 at 4:53 amHi tozé,
The original post had some problems with the quotation marks surrounding the url. I’ve fixed it, but you can try copying and pasting and add in the quotation marks manually or try using the new command above.
August 13th, 2009 at 6:57 amDamon,
Thanks for the quick reply. It was indeed the quotes, it’s working like a charm
August 13th, 2009 at 7:40 amI’m trying the script on XP, but it won’t write the output. Says “Warning: Failed to create the file user_timeline.xml?count=100&page=32″. I tried using a small ‘o’ instead of a capital, but then it seems that the ‘last page wins’, overwriting (instead of appending) whatever file name that I assigned each time a page is downloaded. I’d have to guess the ampersand is throwing it off, how do I work around this?
August 13th, 2009 at 10:17 amHi Joe,
The script above uses a syntax specific to bash on unix systems. Next time I fire up my Windows VM, I can post the proper syntax.
August 13th, 2009 at 12:40 pmIs it possible to download the last 3200 tweets WITHOUT the password? and ONLY the username?
It doesn’t need to be in an XML format. Like .csv would be fine.
Thanks.
August 13th, 2009 at 8:27 pm[...] line lovers can use this clever method to download their tweet XML via cURL. Alternately, web application Backup My Tweets does just that and lets you download your tweets in [...]
August 14th, 2009 at 6:43 amJohn,
Actually, yes you can download anybody’s last 3,200 tweets with the following format.
curl -O "http://twitter.com/statuses/user_timeline/username.xml?count=100&page=[1-32]"
Replace username in the url as desired.
August 15th, 2009 at 8:13 am[...] Adapted from http://dcortesi.com/2009/07/16/for-wil-wheaton/ [...]
August 17th, 2009 at 3:58 pmYou probably want to use https instead of http in the command since it involves your password going out.
August 17th, 2009 at 4:21 pmUpdated with https, although that’s not really much of a concern unless you’re pulling this down in a coffee shop or at a conference. In which case, some of the other Twitter apps aren’t much better.
Thanks for the reminder.
August 17th, 2009 at 9:27 pm[...] line lovers can use this clever method to download their tweet XML via cURL. Alternately, web application Backup My Tweets does just that and lets you download your tweets in [...]
August 22nd, 2009 at 2:10 am[...] you’ve run Damon Cortesi’s handy curl command to download all (or the last 3200) tweets from your twitter account, you’ll have a directory [...]
August 23rd, 2009 at 4:12 am[...] هارد درایو خود داشته باشید. عاشقان خط فرمان می توانند از این روش استفاده کنند. روش دیگر استفاده از سایت Backup My Tweets است. که [...]
August 29th, 2009 at 11:18 am[...] DCortesi . blog » For Wil Wheaton [...]
September 4th, 2009 at 12:04 pm[...] from DCortesi.com, he makes tweetstats.com which is a very cool [...]
September 9th, 2009 at 4:13 pm