-
Notifications
You must be signed in to change notification settings - Fork 287
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How does scattertext generate set of words for plotting, and compute x and y coordinates? #36
Comments
I also have the same question. I'm trying to plot over 800 topics. However around 600 topics actually show up in the plot, and the topic I was looking for didn't show up. Here's my code: topic_feature_builder = st.FeatsFromTopicModel(topic_model) corpus =(st.CorpusFromParsedDocuments(df, len(topic_feature_builder.get_top_model_term_lists()) # 819 topics Is there a way to add the missing topics to the corpus? |
Thanks for the bug report. I appreciate the code sample you've provided, but it's hard to provide support if I can't recreate the issue on my own. Could you make a colab notebook recreating the problem, or post code which can be run independently which reproduces the issue? My initial sense is that some of the topics in may not appear in your data. This could be due to differences in tokenization between the topic model and spaCy's tokenization or simply because that no words from certain topics occurred in your dataset. |
I looked at @jonessarae's data and custom topic model, and it looks like the issue was that |
Hello! I have two questions:
1) How does produce_scattertext_explorer() come up with the set of words to plot?
I've encountered what I think is a discrepancy: After I've set the minimum term frequencies to zero, and the pmi threshold coefficient to zero, produce_scattertext_explorer() generates a set of words that is smaller than the set of words in the corpus TDF.
html = st.produce_scattertext_explorer(corpus,
category='pre',
category_name='China',
not_category_name='Russia',
minimum_term_frequency=0,
minimum_not_category_term_frequency=0,
pmi_threshold_coefficient=0,
return_data=True)
mydata = [t for t in html.items()][1][1]
myterms = [row['term'] for row in mydata]
tdf = corpus.get_term_freq_df()
terms = [t for t in tdf.index]
len(myterms) # 157540
len(terms) # 164689 ... why are these different?
2) How does produce_scattertext_explorer() calculate x and y coordinates?
Would it be possible to add this to the documentation? That would be really, really helpful.
Many thanks, really appreciate your help!
The text was updated successfully, but these errors were encountered: