Our humble apologies for yesterday’s hiccup, which saw thousands of Twitter expense and time entries duplicated in Harvest accounts which were linked to Twitter. We most regret the thousands of emails we delivered to your inboxes. The good news is that all duplicate entries have been rolled back and things are back in order.

Our code depended on Twitter’s API to filter out direct messages that had already been processed by Harvest. As of yesterday afternoon, we no longer depend on Twitter’s filtering options. Going forward, this type of failure in Twitter’s API will not impact Harvest users.

For those who are technical and curious, we’ve decided to get into the details a bit and let you know how it happened, and how we’re preventing it from happening in the future.

We allow communication between Harvest and Twitter via direct messages to the Harvest Twitter account.

Every few minutes, a job retrieves the latest direct messages from Twitter and processes all new ones. Each direct message has an ID associated with it. These ID’s always grow larger over time, so newer direct messages will always have an increasingly higher ID. We keep track of the greatest ID we have retrieved upon every execution.

The Twitter direct message call allows us to pass in a “since ID,” which basically says “give us all direct messages greater than the ID we provide”. We pass along the ID we have stored in our database. Twitter than responds with only the latest and greatest direct messages for processing; messages we have never processed before.

Further, Twitter will only return 20 direct messages at a time. To get the next 20, you must pass a page parameter to the Twitter call. We “smartly” planned for the eventuality that we would be getting > 20 direct messages every few minutes. Basically we looped, making a series of calls to the next page and the next page until the number of direct messages returned from Twitter was < 20. Once we received < 20 direct messages, we knew we had reached the finish line and we were out of pages to retrieve.

Yesterday, around 1 PM EDT, the “since ID” filtering offered by Twitter quit working. So instead of receiving 1 or 2 direct messages from our call to Twitter, we received 20. The looping kicked in, going back in time to get 20 more and 20 more. We were only limited by Twitter’s API throttle as to how far back our job would look for direct messages. With each direct message processed came a new entry in Harvest and a confirmation email.

Every few minutes this would happen again.

We rewrote our job to no longer bother with the since ID. All the looping is still in there, but now we break the loop once the gathered direct messages include the since ID. Then, we filter out all direct messages whose ID is <= to the most recent ID that we had processed. Basically, we’re recreating the since ID logic on our side.

The takeaway is to err on the side of defensive programming when coding against an API. Trust, but verify. While it is not possible to pick up the slack in every situation, if there is something you can accomplish simply on your side of the interaction, it’s probably best to trade a more coding for a simpler, less brittle call to the API you are working with.