Final Update to This Post For those not following along at home, I finally took Twitter Stats to the next level and released a webified version over at TweetStats.com. I, somewhat unfortunately, had to go with TweetStats as twitterstats.com was already taken. I made a post about it here and you can see an example of my stats on the site on the graphs page.

So I’ve been a user on Twitter for a little over a year, but it wasn’t until recently when I hit 2000 tweets that I wanted to see what my Twitter history looked like over that period. Ever being the statistics nerd, I pulled down all of my tweets and using a combination of curl, sed, grep, Excel, and Numbers, managed to generate some nice graphs.

Being the automation weenie that I am, I eventually hacked together a perl script that did everything except paste the data into Numbers.Although I won’t post it here (because I think the Twitterocracy would have a cow with how it’s implemented), you can DM me your email and I’ll send you the code and instructions. See below - bugs be damned, I’ve made it publicly available.

Basically, the script pulls down all your tweets and stores them in an csv file. It then runs some statistics on the csv file and then copies the resulting stats to the OS X clipboard to paste into each table within Numbers. If run with a pre-existing tweets csv file, the script will calculate the different between your current status count and only download the pages necessary, thus saving the Twitter servers from some bandwidth.

For the record, here are mine.

Update Thanks to kosmar for pointing out that I can actually do this entire thing without your password. Head => Wall. I’ll be updating it accordingly and hopefully even making a webservice out of it soon.

In addition to not needing your password, the script should also adjust the times for your tweets to whatever the system time is where you run the script.

Another update: I’ve also posted the script on my site and you can download it here: twitter_stats.zip. Feel free to contact me with any questions via twitter or web.dpc at dcortesi . com.

Many people have noticed a large after-lunch spike around 2pm. At least for me, this was due to Twitter being down most of the morning one day and then tweeting like crazy when it came back online.

BUG FOUND AND SQUASHED

There was a small bug that cropped up after I switched the script to not require your password. It accounts for the odd “[Tuesday|January|2pm] Peak” that people were seeing. This bug has now been fixed and an updated script is available. My apologies.

Unfortunately, if want the most accurate tweets, you will have to rm your csv file and run the script again.

Date::Calc aka failure on line 13
Some of you (on Tiger?) may be missing the Date::Calc module that I use to figure out weekdays. Although I tried to use as few perl modules as possible, this one was essential. Use the following command (thanks to a couple twitter peeps for the reference) to install:
sudo perl -MCPAN -e 'install Date::Calc' and keep hitting ‘y’.

Final (hopefully) Update on this page as it’s getting messy.

For those of you not on OS X with Numbers, there are a few options:
@bck webified my code (w00t): Twitter Stats
@mmc decided to use gnuplot: Twitter Stats in SVG Using GNUPlot
@cbarrett modified it to utilize the Google Chart API: Twitter Stats with GChart
@kejadlen reverse engineered my original script to Ruby: Twitter Stats in Ruby

I still want to write my own webified version (Google Chart aesthetics leave a little bit to be desired…), but I have yet to settle on an option that I like.

I just saw a post about some of the browser capabilities of the new iPhone, and there was one feature that caught my eye:

- new telephone links allows you to integrate phone calls directly from your webpage. remember this is only on safari.

The first thing I thought of was, “Wow, I hope that you can’t somehow execute those links automatically via JavaScript…”. Can you imagine if you browse to a page and your iPhone automatically dials the number of an attacker and listens in on a conversation you might be having? Combine an XSS vulnerability on a high-profile website and a couple of high-profile CEO’s that we _know_ have an iPhone and you could get some pretty interesting dirt!

That would be kind of bad…

Update: Hehe, see.

Awesome - just when I solve the issue of VMWare breaking nmap on OS X, Parallels comes along and does it again. The error is slightly different, however, so the root cause of the problem is likely somewhat different as well.

Starting Nmap 4.20 ( http://insecure.org ) at 2007-06-14 15:07 CDT
getinterfaces: Failed to obtain MAC address for ethernet interface (fw0)
QUITTING!

Thus far, I’ve tried disabling the fw0, en2 and en3 interfaces, with no luck. If I bring down fw0, though (sudo ifconfig fw0 down), I get a different error message similar to the VMWare one.

Starting Nmap 4.20 ( http://insecure.org ) at 2007-06-14 15:11 CDT
getinterfaces: Failed to open ethernet interface (fw0). A possible cause on BSD operating systems is running out of BPF devices (see http://seclists.org/lists/nmap-dev/2006/Jan-Mar/0014.html).
QUITTING!

It seems there’s been a similar problem with Cisco’s VPN software, but the suggested remediation doesn’t work for nmap. I filed a bug report, as I’m sure many others have, so hopefully it will be addressed in a recent update. If I come across a solution, I’ll update this entry…but until then, the only way I can use nmap is by uninstalling Parallels.

Update! After some more detailed information from the Parallels Team, I discovered a way to run nmap successfully. I thought I had tried this approach before, but apparently not. Removing the interface with a `sudo ifconfig fw0 remove` prior to executing nmap seems to allow nmap to run successfully. I seem to have to do this every time as an ip address gets re-assigned to the interface, but it does appear to work!

Update (07/26/2007) The most recent build of Parallels (4560) appears to have fixed the issue metioned above, but another one has manifested itself. Scanning a specific host was able to complete succesfully, but when scanning a network where dead hosts existed would result in a nexthost: failed to determine route error. Specifying the proper interface using the -e parameter seems to address the issue.

There was a recent post on the Nmap Development mailing list about an nmap/vmware osx bug.  I recently experienced the same thing and after brief mention from a co-worker about VMware Fusion’s network interfaces not being “real” interfaces, and confirming this by opening up the Network Preference pane and not having the vmnet interfaces be listed under Network Port Configuration - I dug into the problem a little bit deeper.  The specific error I was getting was the following:

$ sudo nmap –iflist
Starting Nmap 4.20 ( http://insecure.org ) at 2007-05-14 14:09 PDT
getinterfaces: Failed to open ethernet interface (vmnet8). A possible cause on BSD operating systems is running out of BPF devices (see http://seclists.org/lists/nmap-dev/2006/Jan-Mar/0014.html).
QUITTING!

A quick look at my process list for vmware stuff (ps -aux | grep -i vmw) showed the following processes:

root 108 0.0 -0.0 27296 56 ?? Ss Wed01AM 0:00.00 /Library/Application Support/VMware Fusion/vmnet-netifup -d /var/run/vmnet-netif-vmnet8.pid vmnet8 vmnet8
root 111 0.0 -0.0 27296 56 ?? Ss Wed01AM 0:00.00 /Library/Application Support/VMware Fusion/vmnet-netifup -d /var/run/vmnet-netif-vmnet1.pid vmnet1 vmnet1

Ahh, so they aren’t “real” interfaces.  A quick `kill -9 108 111` and Nmap was again able to run successfully.

As previously mentioned, I recently came into the possession of a MacBook pro. Always being the diligent student of security, I decided to enable FileVault for my home directory. Unfortunately, with the combination of Parallels and merging gigs of pcap files, the performance wasn’t really holding up and I decided to revert the process. Relying on the ease of use of Apple’s software, I went into my Security preferences and clicked “Turn off FileVault”. After entering the requisite passwords and dialog boxes, the system logged me out and began the process. A few moments later, I was greeted with this unfortunate message:

An error occurred
An error occurred during decryption (An error occurred during copying ). FileVault will be turned on for this home folder and the home folder will still be encrypted.

An audible sigh escaped my lips as I realized this wasn’t going to be as easy as I hoped. A Google on the error message revealed pathetically few results (2, at the time of this writing…one in Japanese). Some more creative Google searches revealed some useful tips, but trying to repair or restore the filesystem both resulted in failure with cryptic error messages. It seemed the root of the problem was the result of filesystem corruption in the actual encrypted sparseimage that housed my home directory. One of the recommended courses of action included copying the contents of the encrypted image to another directory, deleting the user, and creating a new, replacement user. I really didn’t want to do this…so I came up with another way. These instructions are not verbose and some file permission munging was required, but this should give a general idea of how I solved the problem.

• Create another account
• Login to this second account
• Create a new encrypted sparseimage of larger size than the current one and give it the same password
• Mount both the old sparseimage (located in /Users/<user>) and the new one (~/usernew.sparseimage)
• Copy the files over using tar: `cd /Volumes/<user>; tar cfp - ./* | (cd /Volumes/usernew; tar xfp -)`
• There might be some errors, so that’s why I left the verbose flag out of tar - make sure critical files aren’t missing
• Configure all files on the new filesystem to be owned by the proper user
• Unmount the filesystems and move the new sparseimage to the old one:
• mv /Users/<user>/.sparseimage /Users/<user>/.sparseimage.old && mv ~/usernew.sparseimage /Users/<user>/.sparseimage
• Log out and back in as the original user
• Try to disable FileVault again…it should work this time. :o)

I can already feel my hard drive breathing a sigh of relief.

Holy mac Fanboy, batman! It’s going to be an expensive day for Damon. MacWorld rules supreme!

ohsobeautiful

Dear Steve,

I just wanted to let you know that my “SmartPhone” is going in the trash immediately if the iPhone is as good as it looks.

Thank you,

DCortesi.com

I’m frequently firing up tcpdump on my mac, but then having to restart it with the correct interface if I happened to be on a wireless network (which I frequently am on the mac). I finally got tired of that, so I whipped up a quick script that determines which interface is up and consequently starts tcpdump on that interface.  Allow me to `cat /usr/bin/stcpdump` for you:

#!/bin/sh

INT=`ifconfig | grep -A3 "en.:" | grep -B2 inet\ | head -n 1 | cut -f1 -d:`
sudo tcpdump -i $INT $@

Much better - saves me a big of grief. ;o)