[GitHub] Service timeout after requesting webhook test payload

[Poorchop]

I can't receive any notifications from GitHub whatsoever via webhooks, as my bot won't even accept the initial test payload. Here is the traceback:

INFO 2014-09-07T02:54:26 Got ping event from GitHub.
INFO 2014-09-07T02:54:31 HTTP request: 192.30.252.41 - "POST /github HTTP/1.1"
     200 -
ERROR 2014-09-07T02:54:31 Uncaught exception in GithubCallback.doPost:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/supybot/log.py", line 361, in m
    return f(self, *args, **kwargs)
  File "/home/poorchop/runbot/plugins/GitHub/plugin.py", line 137, in doPost
    self.send_response(200)
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 393, in send_response
    (self.protocol_version, code, message))
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
ERROR 2014-09-07T02:54:31 Exception id: 0x51f87
----------------------------------------
Exception happened during processing of request from ('192.30.252.41', 54738)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 704, in finish
    self.wfile.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------

I'm using the latest version of the GitHub plugin. The webserver at the port I have specified loads up properly, and the GitHub plugin itself loads without any issues. I'm running this on Debian 7.

[Mikaela]

Are you using the latest version of the plugin and what do you see at GitHub?

[Poorchop]

Are you using the latest version of the plugin

Yes, I am using the latest version of the plugin.

and what do you see at GitHub?

Under the "Recent Deliveries" section, I see a red triangle with an exclamation mark inside next to the name of the payload, as well as the date and time GitHub attempted to send the payload and the words "We couldn’t deliver this payload: Service Timeout." There is an option to redilver the payload, which I have tried multiple times, but I get a service timeout every single time I request another test payload.

[Mikaela]

When you go to "manage webhook", what are all settings you see there and their values?

When you click some webhook, what do you see there in "request" and "response" tabs?

Also, what version of Limnoria?

[Poorchop]

When you go to "manage webhook", what are all settings you see there and their values?

Payload URL: http://mysite.com:7410/github (obviously using my real domain name, not mysite.com)
Content type: application/x-www-form-urlencoded
Secret:
Which events would you like to trigger this webhook?: Send me everything.

• Active: We will deliver event details when this hook is triggered.

When you click some webhook, what do you see there in "request" and "response" tabs?

I'm not sure what you mean by this. Where can I click on a webhook? I just have one button to redeliver the test payload, and I don't see any "request" or "response" tabs. According to a GitHub employee, they are seeing a 504 Gateway Timeout from their end.

Also, what version of Limnoria?

The current (running) version of this Supybot is 0.83.4.1+limnoria 2014.06.26, running on Python 2.7.3 (default, Mar 14 2014, 17:55:54) [GCC 4.6.3].

[Mikaela]

Your settings look correct. I am not just sure about your hook address as I cannot test that it works for me as you aren't giving it.

Recent deliveries has all deliveries and they can be clicked for me.

Your Limnoria doesn't sem too old. How about the GitHub plugin? If you have git cloned it, could you try git pull or with PluginDownloader plugindownloader install ProgVal GitHub and reload GitHub?

And last, do you have any configuration variables set? They were empty in older versions and if they are empty, you can use config setdefault plugins.github. for them and try if it helps.

If nothing of these works, I don't know anything that could solve this and this is probably real issue.

[Poorchop]

I can PM you the real hook address on IRC provided you don't have an issue with people sending you PMs. When I click on the webhook in recent deliveries, it just expands to show the service timeout error message and the redeliver button.

I just updated Limnoria, both the dependencies and the bot itself. I unloaded the GitHub plugin that I installed from the master branch, I deleted the plugin from ~/poorchop/runbot/plugins, I reinstalled with plugindownloader, and I did a reload GitHub. Finally, I did a config setdefault plugins.github. The plugin installed correctly and all operations succeeded without any errors. I tried the test payload once more, but I'm still getting a service timeout.

[Mikaela]

I can PM you the real hook address on IRC provided you don't have an issue with people sending you PMs.

I am Mikaela at IRC and I don't have issues with PMs which are in language I understand, have content and have common sense unlike some others.

You must do config setdefault for all variables there. See also documentation on configuration.

[Poorchop]

Here is another error I found. I'm not positive, but this might be what resulted after a GitHub employee tried with a generic curl POST at the payload URL:

INFO 2014-09-07T12:14:51 HTTP request: 192.30.252.38 - "POST /github HTTP/1.1"
     200 -
ERROR 2014-09-07T12:14:51 Uncaught exception in GithubCallback.doPost:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/supybot/log.py", line 361, in m
    return f(self, *args, **kwargs)
  File "/home/poorchop/runbot/plugins/GitHub/plugin.py", line 134, in doPost
    event = headers['x-github-event']
KeyError: 'x-github-event'
ERROR 2014-09-07T12:14:51 Exception id: 0x2de26 

[Mikaela]

@progval , ping.

I can access the web interface and it has only IPv4 address (GitHub is IPv4-only) and the bot seems to be listening on it. Settings also look correct looking at previous comments.

[Poorchop]

In case this is of any help, I contacted GitHub support and one of the employees has been troubleshooting this for me. Here is part of what he had to say about the issue:

I just tried making a request manually from one of our machines, and that went through with no error (see curl -v output below). However, I did notice that it took extremely long for the request to be processed -- over 15 seconds (for 2 bytes of data).

[progval]

It works fine for me. I think your bot/computer/connection is just slow

[Mikaela]

It works fine for me. I think your bot/computer/connection is just slow

I can access the web interface without slowness and @Poorchop (PurpleDog at IRC) told that the bot reported latency as 0.26 seconds.

[Poorchop]

I pushed a commit to the repo with the webhook, and my bot prints the events to my channel, so it appears like it's working even though GitHub still says that the payloads are timing out. As for my the bot being slow, that doesn't really make sense because I'm running the same bot as everyone else. My server is more than capable of handling Limnoria with plenty of resources to spare, and both my upload and download are ~20 Mbit/sec.

Assuming any of these components was actually slow, I still don't understand why I would be seeing the error messages that I posted.

[ghost]

I have a similar issue, but it only happens since I made some modifications to the plugin that I plan to pull request later on.

There is no error in my logs; the bot announces all the commits, but with a 10-second delay between them. protocols.irc.throttleTime is set to 1.0. The latency is 0.07 seconds.

I just confirmed that the issue does not occur for me without my modifications.

[its-saurabhjain]

Similar timeout issue while integrating with Jenkins and Github. I used Jenkins GitHub plugin and configured to manage the webhook. However when i push the code to Github , getting the same service timeout issue...looking for a resolution

[progval]

@its-saurabhjain This repo contains plugins for the Supybot/Limnoria IRC bot. Is this what you are having issues with?