Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing 2017 and 2018 schedule data #298

Open
Couragyn opened this issue May 7, 2018 · 17 comments
Open

Missing 2017 and 2018 schedule data #298

Couragyn opened this issue May 7, 2018 · 17 comments

Comments

@Couragyn
Copy link

Couragyn commented May 7, 2018

I am missing schedule dates from 2017 up. I have seen this issue elsewhere, but the solutions suggested are not working for me as these files are not in scripts:

  1. update_sched.py --year 2017
  2. nflgame-update-schedule --year 2017

nfldb-update has a '--update-schedules' flag, but it does not add in the missing years.

Any ideas?

@matthewcampbell10
Copy link

Did you find a solution for this? When i update i am not getting any 2017 data?

@Couragyn
Copy link
Author

Couragyn commented Jul 9, 2018

I have not yet found a solution

@ochawkeye
Copy link
Contributor

I believe my fork of nflgame has all of the games played in the regular and postseason of 2017 included.
https://github.com/ochawkeye/nflgame

No schedule has yet been posted to nfl.com for the 2018 season
http://www.nfl.com/liveupdate/scorestrip/ss.xml

@matthewcampbell10
Copy link

ochawkeye...how do i implement your fork? I am use to just running the nfldb-update script and the most recent data is pulled. This looks like something different?

@matthewcampbell10
Copy link

matthewcampbell10 commented Jul 9, 2018

If you know of a solution to keep nfldb-udpate working, that would be ideal. I cant seem to figure out why when i ran it last year i was getting 2017 data and now I only get 2016 data. I did have to reinstall everything on a new machine so wondering if i just did that piece wrong...but doesnt seem like it

@ochawkeye
Copy link
Contributor

Wellllll....I'm guessing that you're still running the original install from PyPI that hasn't been updated in ages.

If you are, then you can go one of two routes...

  1. You can uninstall and re-install using instead the (also un-maintained github repository gulp) via Updating for 2017 Data #296 instructions.
  2. You can just try copying and pasting my schedule.json into your /nflgame folder and try running your same nfldb-update script. That might succeed in downloading your missing games.

@matthewcampbell10
Copy link

Doesnt look like the schedule.json solution worked. This is what i get when running nfldb if that might help? As you can tell i am not a developer :)

STARTING NFLDB UPDATE AT 2018-07-09 16:10:38.037000
Connecting to nfldb... done.
Setting timezone to UTC... done.
Locking write access to tables... done.
Updating season phase, year and week... done.
Adding schedule data for 16 games... done.
Bulk inserting data for 273 games...
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Traceback (most recent call last):
File "nfldb-update", line 39, in
nfldb.update.run(**vars(args))
File "C:\Python27\lib\site-packages\nfldb\update.py", line 535, in run
doit()
File "C:\Python27\lib\site-packages\nfldb\update.py", line 525, in doit
update_games(db, batch_size=batch_size)
File "C:\Python27\lib\site-packages\nfldb\update.py", line 397, in update_games
bulk_insert_game_data(cursor, scheduled, batch_size=batch_size)
File "C:\Python27\lib\site-packages\nfldb\update.py", line 195, in bulk_insert_game_data
do()
File "C:\Python27\lib\site-packages\nfldb\update.py", line 188, in do
nfldb.db._big_insert(cursor, table, bulk[table])
File "C:\Python27\lib\site-packages\nfldb\db.py", line 356, in _big_insert
% (table, insert_fields, values))
File "C:\Python27\lib\site-packages\psycopg2\extras.py", line 249, in execute
return super(RealDictCursor, self).execute(query, vars)
psycopg2.IntegrityError: insert or update on table "play" violates foreign key constraint "play_pos_team_fkey"
DETAIL: Key (pos_team)=(JAX) is not present in table "team".

@matthewcampbell10
Copy link

With the second solution i get the following response. I didn't think git was something that needed to be installed...at least this isnt described in any of the tutorials i have gone through

C:\Users\matth>pip install git+https://github.com/BurntSushi/nflgame.git
Collecting git+https://github.com/BurntSushi/nflgame.git
Cloning https://github.com/BurntSushi/nflgame.git to c:\users\matth\appdata\local\temp\pip-req-build-swe2sb
Error [Error 2] The system cannot find the file specified while executing command git clone -q https://github.com/BurntSushi/nflgame.git c:\users\matth\appdata\local\temp\pip-req-build-swe2sb
Cannot find command 'git' - do you have 'git' installed and in your PATH?

@ochawkeye
Copy link
Contributor

The JAX not being present in the table is not related to the schedule not updating.

See #252 for related discussion on that.

For your second issue...Yes, if you aren't able to git clone then you won't be able to pip install from github directly.

@matthewcampbell10
Copy link

Interesting so it looks like it added a ton more games...but i am still not getting 2017?

STARTING NFLDB UPDATE AT 2018-07-09 16:29:28.216000
Connecting to nfldb... done.
Setting timezone to UTC... done.
Locking write access to tables... done.
Updating season phase, year and week... done.
Adding schedule data for 339 games... done.
Bulk inserting data for 596 games...
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Traceback (most recent call last):
File "nfldb-update", line 39, in
nfldb.update.run(**vars(args))
File "C:\Python27\lib\site-packages\nfldb\update.py", line 535, in run
doit()
File "C:\Python27\lib\site-packages\nfldb\update.py", line 525, in doit
update_games(db, batch_size=batch_size)
File "C:\Python27\lib\site-packages\nfldb\update.py", line 397, in update_games
bulk_insert_game_data(cursor, scheduled, batch_size=batch_size)
File "C:\Python27\lib\site-packages\nfldb\update.py", line 195, in bulk_insert_game_data
do()
File "C:\Python27\lib\site-packages\nfldb\update.py", line 188, in do
nfldb.db._big_insert(cursor, table, bulk[table])
File "C:\Python27\lib\site-packages\nfldb\db.py", line 356, in _big_insert
% (table, insert_fields, values))
File "C:\Python27\lib\site-packages\psycopg2\extras.py", line 249, in execute
return super(RealDictCursor, self).execute(query, vars)
psycopg2.IntegrityError: insert or update on table "play" violates foreign key constraint "play_pos_team_fkey"
DETAIL: Key (pos_team)=(JAX) is not present in table "team".

@matthewcampbell10
Copy link

This was after installing git and then updating pip install git+https://github.com/BurntSushi/nflgame.git

I feel like i am close but something is still missing

@matthewcampbell10
Copy link

ochawkeye...would you be interested in working on a custom solution for me? Can work out payment arrangements of some sort. Let me know.

Thanks

@ochawkeye
Copy link
Contributor

I'm not interested in taking payment, no, but I'm happy to assist in getting you going. Would be good to see what a semi-clean install looks like so I could take some notes to maybe get something documented to assist others in the future.

If you're comfortable with me temporarily accessing your system, run TeamViewer QuickStart and contact me at benjaminDOTjDOTschroederATgmailDOTcom.

I believe your schedule & games are now updating properly but you've still got an issue with the JAC/JAX changeover that happened a couple of years ago since your traceback states DETAIL: Key (pos_team)=(JAX) is not present in table "team".

@andymiller-og
Copy link

Funny, I had preseason data in there from when I ran it last year in late august. Today, when i first went back in and reran the nfldb-update, I got week 17 but that was it.
image

I then ran nfdb-update --update-schedules. Took some time but didnt do anything!
image

Anyone have any other ideas?

@andymiller-og
Copy link

So I ran python update_sched.py --year 2017 from the nflgame directory. This updated the schedule.json file with the appropriate games. Then i ran nfldb-update and the games updated:
image

@ghost
Copy link

ghost commented Jul 26, 2018

Any updates on getting the 2018 players roster?
When I run:
import nfldb
db = nfldb.connect()
q = nfldb.Query(db)
q.game(season_year=2017, season_type='Regular')
for pp in q.as_aggregate():
if str(pp.player.position) in ['QB', 'RB', 'WR', 'TE', 'K']:
print pp.player

I get all the players coming in. But when I run it with 2018, 'Regular' or 'Preseason'
Nothing returns...

-Front-End developer lost in the back-end world!

@ochawkeye
Copy link
Contributor

There's not anything for you to get yet as far as a player roster. The query you performed above is aggregating statistics compiled over the course of the 2017 regular season. There haven't been any plays yet run in the 2018 regular or preseason.

I succeeded in updating my schedule today and can see the games with nfldb but there are no players that have aggregated any stats yet for the season like you're looking for.

import nfldb

db = nfldb.connect()
q = nfldb.Query(db)
q.game(season_year=2018, season_type='Regular', week=1)

for game in q.as_games():
    print game

print 'v'*25
for pp in q.as_aggregate():
    print pp.player
print '^'*25
Regular 2018 week 1 on 09/10 at 09:20PM, LA (0) at OAK (0)
Regular 2018 week 1 on 09/10 at 06:10PM, NYJ (0) at DET (0)
Regular 2018 week 1 on 09/09 at 07:20PM, CHI (0) at GB (0)
Regular 2018 week 1 on 09/09 at 03:25PM, SEA (0) at DEN (0)
Regular 2018 week 1 on 09/09 at 03:25PM, DAL (0) at CAR (0)
Regular 2018 week 1 on 09/09 at 03:25PM, WAS (0) at ARI (0)
Regular 2018 week 1 on 09/09 at 03:05PM, KC (0) at UNK (0)
Regular 2018 week 1 on 09/09 at 12:00PM, JAC (0) at NYG (0)
Regular 2018 week 1 on 09/09 at 12:00PM, TB (0) at NO (0)
Regular 2018 week 1 on 09/09 at 12:00PM, HOU (0) at NE (0)
Regular 2018 week 1 on 09/09 at 12:00PM, SF (0) at MIN (0)
Regular 2018 week 1 on 09/09 at 12:00PM, TEN (0) at MIA (0)
Regular 2018 week 1 on 09/09 at 12:00PM, CIN (0) at IND (0)
Regular 2018 week 1 on 09/09 at 12:00PM, PIT (0) at CLE (0)
Regular 2018 week 1 on 09/09 at 12:00PM, BUF (0) at BAL (0)
Regular 2018 week 1 on 09/06 at 07:20PM, ATL (0) at PHI (0)
vvvvvvvvvvvvvvvvvvvvvvvvv
^^^^^^^^^^^^^^^^^^^^^^^^^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants