Skip to content

killakam3084/anagrams

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

18 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Tweet Anagram Finder


A program to mimic the behavior of https://twitter.com/anagramatron

anagrams/ contains:

  • a file which parses input tweet data -- anagrams.py
  • a file to make dummy tweet data given a dictionary -- make_tweet.json

dictionaries/ contains:

  • a small dictionary of the 1,000 most common words common_word_dictionary.txt
  • a large dictionary of ~ 170,000 words -- dictionary.txt
  • these are for passing to make_tweet_json.py to construct tweet data

tweet_data/ contains:

  • a small .json file of sample tweet data -- tweets.json

bin/

  • the binaries to execute; added to path with pip install

To execute

  1. Clone repository: git clone https://github.com/killakam3084/anagrams.git

  2. Install anagrams package cd anagrams/ && pip install .

  3. To make sample JSON make-tweet-data -l 3 -n 100000 tweets.json dictionary.txt

     usage: make-tweet-data [-h] [-l, word-length WORD_LENGTH]
             	       [-n, number-of-tweets NUM_TWEETS]
                    	output_file dictionary_file
    
     		A program to make dummy tweet json for anagrams
    
     positional arguments:
     output_file           A output file to write tweet data to.
     dictionary_file       The dictionary to construct anagrams.
    
     optional arguments:
     -h, --help            show this help message and exit
     -l, word-length WORD_LENGTH
                     The max character count for a word.
     -n, number-of-tweets NUM_TWEETS
                     The number of tweets to constructs.
    
  4. Run script

    usage: find-anagrams [-h] input_file
    		A vanilla program to detect anagrams in tweet json
    
    positional arguments:
    input_file  The input file of tweets.
    
    optional arguments:
    -h, --help  show this help message and exit
    
    Certainly this isn't how Anagramatron does it	
    

Notes on ijjson iterative parser**

  • ijson is an iterative JSON parser with a standard Python iterator interface
  • Ijson provides several implementations of the actual parsing in the form of backends located in ijson/backends:
  • The script should output memory_profiler data showing the effectiveness of ijson's lazy loading
  • Processed 50,000,000 two word tweets with words of length 4 in about ~ 2 hours, around ~3.5G file size.

Releases

No releases published

Packages

No packages published

Languages