Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming does not stop #106

Open
tanroopdhillon opened this issue Jun 12, 2017 · 2 comments
Open

Streaming does not stop #106

tanroopdhillon opened this issue Jun 12, 2017 · 2 comments

Comments

@tanroopdhillon
Copy link

This is my configuration:
StreamingContext streamingContext = new StreamingContext(sparkcontext, Durations.seconds(60000));

But the streaming doesn't seem to be stopping. I keep getting messages and is never able to move ahead

@jniebuhr
Copy link

What you specify there is your window duration. A window duration of 60000 seconds means it will gather messages for 60000 seconds (so, 41 days) and then start processing them. Then it'll gather messages for another 60000 messages.
I would recommend using a lower time window that's suitable for your application and processing flow.

@compae
Copy link
Member

compae commented Aug 18, 2017

I'm agree with @jniebuhr , but it's possible that we have one bug in the streamingContext stop function.

Could you add more info or logs??

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants