Cleanup identification of github URLs

[npch]

Currently, we grab all github urls including blobs and duplicates. We should remove these (unless we are trying to test for stickiness of blobs).

[andreww]

I think we should probably keep these, and do some "identification" of link type early in the processing workflow. It may be useful to know how many links are to blobs, and how many are to the base repository (or to a user page), for example.

[npch]

Commit efecff0 cleans up duplicate URLs.

Adding an identification of the link type shouldn't be a problem, but will break the data structure we are currently using.