Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Distributed / serverless and online operation? #2

Open
andrewluetgers opened this issue Aug 11, 2021 · 0 comments
Open

Distributed / serverless and online operation? #2

andrewluetgers opened this issue Aug 11, 2021 · 0 comments

Comments

@andrewluetgers
Copy link

andrewluetgers commented Aug 11, 2021

Very interesting work. When I first learned of UMAP and HDBSCAN I intuitively was curious of the whole bert embeddings -> umap -> hdbscan -> topics approach. I am quite unsurprised at the effectiveness of this approach. It is not intuitive that your approach would achieve similar results and generalizability. Congratulations this appears to be a meaningful contribution especially as it looks to be much cheaper computationally.

That cost aspect of things is really important for large scale online uses. I am working on a document classification service and there are two operational requirements for the service to be optimal in terms of operating expense.
First it needs to be online. As I add new documents the topic model should adjust. Thus there is no need to reprocess all documents to keep the model up to date as the corpus grows.
Second it needs to be distributable such that I can farm out the work of any given document or batch of documents to a serverless function, and the lighter these functions are the better.

Your approach seems like it is far more amenable to these constraints, I'd love to explore them a bit. What is your advice on this and your sense of what would be required to implement online and distributed operation of this algorithm?

@andrewluetgers andrewluetgers changed the title Two aspects of variation, distribution / serverless and online updates? Distributed / serverless and online operation? Aug 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant