Skip to content

Home

Jump to bottom Edit New page
rhubarb edited this page Sep 13, 2010 · 2 revisions

bankjob

Bankjob is a command line ruby program for scraping online banking sites and producing statements as local files in OFX (Open Fincancial Exchange) or CSV (Comma Separated Values) formats.

Bankjob was created for people like me who want to get their bank data into a 3rd party application but whose bank does not support downloads in OFX format.
It’s also useful for keeping a permanent store of bank statements on your computer for reading in Excel.

Installing

Mac OSX / Linux:

sudo gem install bankjob

Windows:

gem install bankjob

The basics

  • Scrapes an online banking website to produce a bank statement
  • Stores bank statements locally in CSV files, which can be loaded directly in spreadsheets like Microsoft Excel
  • Stores bank statements locally in OFX files, which can be imported by many programs such as Quicken, MS Money, Gnu Cash and uploaded to some web applications
  • Built-in support for uploading to your Wesabe account (www.wesabe.com)
  • Supports coding of simple rules in ruby for modifying transaction details. E.g. automatically change “pment inst 3245003” to “paid home loan interest”

Usage

bankjob --csv c:\bank\csv --scraper c:\bank\my_bpi_scraper.rb
        --scraper-args "<my_username> <my_password>"
        --wesabe "<wesabe_user> <wesabe_pass> <wesabe_acct>
        --log c:\bank\bankjob.log --debug

I have this command in a .bat file which is launched daily by a scheduled task on my windows Media Center PC (which, since it’s always on and connected to the internet, makes a useful home server)

This one command will:

  • scrape my online banking website after logging in as me and navigating to the page with recent transactions
  • apply some rules, coded in the my_bpi_scraper.rb file that make the descriptions more readable
  • produce a statement in comma-separated-value format, keeping the original raw data as well as the new descriptions,
    storing that in a file with a name like 20090327-20090406.csv in my local directory c:\bank\csv (serves as a sort of permanent ‘audit’ record)
  • produce an OFX document with the same statement information
  • upload the OFX statement to my wesabe account
  • log debug-level details in bankjob.log

Use bankjob -h to see all of the options.

Wesabe

Wesabe (www.wesabe.com) is a sort of free web.2.0-money-manager-financial-social-networking application that I’m using to track my bank acounts.
I like it better than my bank’s online banking apps because it:

  • shows me all my accounts at once
  • lets me track certain kinds of spending, with automatically applied tags and nice graphs
  • has an iphone-web-app (and a non-iphone one too)
  • even has a nifty dashboard widget for Mac OS X.
  • does much more than this but you should go to their site to find out more.

At this point I should point out that Bankjob and its developers (okay developer – there’s just me, I admit it. God it’s lonely here) have no affiliation with Wesabe whatsoever. I just happen to use Wesabe and like what it has to offer.
In fact I just happened to want to use Wesabe, but couldn’t because my bank doesn’t offer OFX downloads and is not supported by the automatic updater. Hence I developed Bankjob to allow me to use Wesabe.

I think there are a lot of other non-US would-be Wesabe users out there in the same boat: Wesabe and its brethren do not support our banks, and our banks don’t make it any easier by supporting OFX downloads.

There are similar services out there like Mint, Quicken Online, Yodlee and others (see one comparison here), or just google them.
The problem with most of these is that they offer even less support than Wesabe for banks outside of the US. Wesabe supports some non-US banks, but more importantly offers a public API which allows apps like Bankjob to upload accounts – among other things. (Actually I think Yodlee might have more support for non-US banks but still no public API – if your Bank is not officially supported you’re out of luck)

To use the Wesabe upload feature of Bankjob you need a Wesabe username and password. If you have more than one account with Wesabe you need to know the ID that Wesabe gives the account. This is easy to find out: if you have one a account it’s “1” and you don’t need to specify it. If you have more than one account, Bankjob will help you out. Use:
bankjob --wesabe-help
to see the general help information, but to test your Wesabe account, use
bankjob --wesabe-help "<wesabe-user-name> <wesabe-password>"
(you’ll need to use the quotes, with a space between the username and password)

With this command bankjob will list out all of your Wesabe accounts and tell you how to upload to them.

I have only one account at the moment, and here’s what happens when I use --wesabe-help (the text in angle brackets is added by me to protect my account, but otherwise this is real input/output)

[me:~] bankjob --wesabe-help "<my-wesabe-user> <my-wesabe-password>"
Connecting to Wesabe...
You have 1 Wesabe accounts:
 Account Name: <The name of my bank account in wesabe>
    wesabe id: 1
   account no: 0001
         type: Checking
      balance: 2196.12
         bank: <The name of my bank>
To upload to this account use:
  bankjob [other bankjob args] --wesabe "<my-wesabe-user> <my-wesabe-password> 1"

Since you have one account you do not need to specify the id number, use:
  bankjob [other bankjob args] --wesabe "<my-wesabe-user> <my-wesabe-password>"

Again, if in any doubt bankjob --wesabe-help should set you straight.

Rubydoc

http://bankjob.rubyforge.org/rdoc

Forum

http://groups.google.com/group/bankjob

Creating your own Scraper

Unfortunately Bankjob isn’t much good to you in its natural state unless you happen to bank with the same bank as me.
To make it useful you need a scraper for your own online bank site. How hard that will be depends on how handy you are with Ruby programming and with using Hpricot and Mechanize

You’ll get some help from:

  • The Mechanize docs
  • The Hpricot wiki
  • The BpiScraper is included in the scrapers directory and is well documented as an example.
  • The BaseScraper is the superclass of BpiScraper. Using it as a base class for your own scraper is optional, but it saves you a little work by starting up Mechanize for you.
  • The Scraper class itself, which you must subclass with your own Scraper, has more to say on creating your own scraper in its rubydoc
  • Look for help on the forum
  • Ask me directly rhubarb

Warning: Don’t upload or email anyone (including me) your private online banking information. If you create your own scraper, please share it, but remove any account details from it first.

If and when you do develop a scraper for your banksite, send it to me (without any passwords or private banking info of course!) and I’ll add it to the built-in scrapers.

How to submit patches

You can fetch the source from github at:

http://github.com/rhubarb/bankjob/tree/master

using:

git clone git://github.com/rhubarb/bankjob.git

and submit any patches to the forum.

License

This code is free to use under the terms of the MIT license.

As far as rhubarb is concerned, this means you can:
- Use it in your private projects
- Use it in your public projects
- Use it in your commercial projects and make big bucks
- Mulch it up and use it in your garden as fertilizer
- Generally go crazy with it

Contact

Comments are welcome as are feature requests and patches.

Send an email to rhubarb
Or check out the forum

Add a custom footer
Add a custom sidebar
Clone this wiki locally