Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add workflow to gen data #362

Open
wants to merge 299 commits into
base: feat/pull-v2-api
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
299 commits
Select commit Hold shift + click to select a range
ac64ba0
separate out step to check postgres so that we can easily check the
ChenglimEar Nov 1, 2023
f08c7c1
remove unused script
ChenglimEar Nov 1, 2023
8b55cd9
remove new version of sqlalchemy since it breaks csvsql in `make import`
ChenglimEar Nov 1, 2023
d239c1f
add some conditions for job execution in main workflow
ChenglimEar Nov 1, 2023
9a64554
add check for version of sqlalchemy
ChenglimEar Nov 1, 2023
1ee8282
get container to rebuild when requirements change
ChenglimEar Nov 1, 2023
6ca18db
adjust logic for when jobs run in main workflow
ChenglimEar Nov 1, 2023
aacdec8
try summarizing tables created in main workflow
ChenglimEar Nov 1, 2023
3e11a12
make sure we clean up downloads before we start
ChenglimEar Nov 1, 2023
b883c21
Merge branch 'master' into add-workflow-to-gen-data
ChenglimEar Nov 1, 2023
565aede
try testing csvsql early
ChenglimEar Nov 1, 2023
a80b6e7
some logging to test csvsql
ChenglimEar Nov 1, 2023
c9988aa
turn on verbose output while testing csvsql
ChenglimEar Nov 1, 2023
0dd3c03
try sending csv file through stdin for csvsql
ChenglimEar Nov 1, 2023
9678551
upgrade csvkit to 1.3.0 and upgraded its dependencies where needed
ChenglimEar Nov 2, 2023
a1812ff
remove files no longer needed in download dir
ChenglimEar Nov 2, 2023
062d4b1
set postgres version in dev container and workflow to 9.6 to match
ChenglimEar Nov 2, 2023
82408d9
Merge branches 'feat/pull-v2-api' and 'feat/pull-v2-api' of github.co…
ChenglimEar Nov 2, 2023
cf785f2
Merge branch 'feat/pull-v2-api' into add-workflow-to-gen-data
ChenglimEar Nov 2, 2023
db8c504
update workflow names
ChenglimEar Nov 2, 2023
c176028
make use of sql files to create tables
ChenglimEar Nov 2, 2023
48f6259
Update import-file to display schema of created table
ChenglimEar Nov 2, 2023
ad869ef
Update import-file to log more info about postgresql tables
ChenglimEar Nov 2, 2023
dde12b4
Update import-file to point psql to DATABASE_NAME
ChenglimEar Nov 2, 2023
c33d743
Update import-file to use the right quote around table name in psql
ChenglimEar Nov 2, 2023
f9f2654
Update import-file to remove debug logging
ChenglimEar Nov 2, 2023
23ff55b
Update Makefile to use saved sql for creating tables from spreadsheet…
ChenglimEar Nov 2, 2023
84c3a26
fix Makefile by moving bash into file and saved generated sql for tab…
ChenglimEar Nov 3, 2023
a51ee6c
some fixes to get csvkit 1.3.0 working - not fully working yet...
ChenglimEar Nov 3, 2023
bb4354b
Merge branch 'upgrade-csvkit' of github.com:caciviclab/disclosure-bac…
ChenglimEar Nov 3, 2023
dea94a1
make sure data upload for spreadsheet data does not use inference (ie
ChenglimEar Nov 3, 2023
ac93903
debug version of csvkit installed
ChenglimEar Nov 3, 2023
fb456c7
verify python version at time of install on travis
ChenglimEar Nov 3, 2023
929372a
remove sudo for pip install
ChenglimEar Nov 3, 2023
b6dfe8e
Merge branch 'upgrade-csvkit' into add-workflow-to-gen-data
ChenglimEar Nov 3, 2023
ff5adfe
remove download/main.py dependency on latest version of sqlalchemy
ChenglimEar Nov 3, 2023
a2e99a0
use later postgres
ChenglimEar Nov 3, 2023
8394fa3
update postgres for dev container also
ChenglimEar Nov 3, 2023
809710d
download new netfile csvs before import
ChenglimEar Nov 3, 2023
f7cf802
gracefully handle records missing transaction data
ChenglimEar Nov 3, 2023
2e0387d
add netfile v2 data to database during import
ChenglimEar Nov 3, 2023
b6b668e
make sure dir exists for saving v2 csv files
ChenglimEar Nov 3, 2023
f496e73
forgot to import os
ChenglimEar Nov 3, 2023
2eb24c6
fix param name
ChenglimEar Nov 3, 2023
c2a0f07
make netfile v2 download a part of `make download`
ChenglimEar Nov 3, 2023
715d296
add requirements for netfile v2 code
ChenglimEar Nov 4, 2023
81d7d1b
update python-dateutil
ChenglimEar Nov 4, 2023
9ff8b0e
try to cause failure when pip install fails
ChenglimEar Nov 4, 2023
6c9dcbf
upgrade babel
ChenglimEar Nov 4, 2023
80e07e3
update pytz
ChenglimEar Nov 4, 2023
8f42b3d
Merge branch 'check-using-digest' into upgrade-csvkit
ChenglimEar Nov 9, 2023
a1bfe9a
Merge branch 'upgrade-csvkit' into add-workflow-to-gen-data
ChenglimEar Nov 9, 2023
e297bd6
Merge branch 'check-using-digest' into upgrade-csvkit
ChenglimEar Nov 12, 2023
fb60d32
Merge branch 'check-using-digest' into upgrade-csvkit
ChenglimEar Nov 12, 2023
b2fb864
allow csvkit to pull in the correct agate dependencies and add script to
ChenglimEar Nov 13, 2023
3c68082
remove whitespace for some key columns
ChenglimEar Nov 13, 2023
05cee7c
Merge branch 'check-using-digest' into upgrade-csvkit
ChenglimEar Nov 13, 2023
fb33f7b
Merge branch 'upgrade-csvkit' into add-workflow-to-gen-data
ChenglimEar Nov 13, 2023
8076382
split contributions by type to multiple elections when a candidate was
ChenglimEar Nov 18, 2023
ea0c9ab
removed commented code
ChenglimEar Nov 18, 2023
b6937a3
Merge branch 'fix-contributions-by-type' into upgrade-csvkit
ChenglimEar Nov 18, 2023
b3e95b7
create candidate_summary view to associate "Summary" info with specific
ChenglimEar Nov 19, 2023
7cc4317
add total contributions to digest.json
ChenglimEar Nov 20, 2023
c079642
Merge branch 'add-totals-to-digests' into fix-contributions-by-type
ChenglimEar Nov 20, 2023
bb6c874
use hash of hash for contributions by type
ChenglimEar Nov 25, 2023
a460ff8
add total contributions by type and source to digests
ChenglimEar Nov 25, 2023
7c707ec
take election into account when calculating total contributions and
ChenglimEar Nov 25, 2023
3c9251e
organize totals calculated from various sources in digests.json
ChenglimEar Nov 25, 2023
248e6bd
update digests.json to include more totals
ChenglimEar Nov 25, 2023
ea1a077
Merge branch 'add-totals-to-digests' into fix-contributions-by-type
ChenglimEar Nov 25, 2023
dc829b0
calculate contribution totals for all tickets (candidates and referen…
ChenglimEar Nov 25, 2023
ee2e56d
Merge branch 'add-totals-to-digests' into fix-contributions-by-type
ChenglimEar Nov 25, 2023
1cf25fa
add more totals to digest and separate by contributions vs expenditures
ChenglimEar Nov 28, 2023
c8c2d9a
Merge branch 'add-totals-to-digests' into fix-contributions-by-type
ChenglimEar Nov 28, 2023
020eb5b
update expenditures to be split on election and other calculations to
ChenglimEar Nov 28, 2023
e32d6e1
Merge branch 'master' into add-totals-to-digests
ChenglimEar Nov 28, 2023
e4a9a67
Merge branch 'add-totals-to-digests' into fix-contributions-by-type
ChenglimEar Nov 28, 2023
82ededc
revert committee contribution list calculator
ChenglimEar Nov 30, 2023
14f67b4
some comments about the totals calculated for digests.json
ChenglimEar Nov 30, 2023
4ecd440
Merge branch 'add-totals-to-digests' into fix-contributions-by-type
ChenglimEar Nov 30, 2023
dca1bae
Merge branch 'master' into add-totals-to-digests
ChenglimEar Nov 30, 2023
c0bbc32
Merge branch 'add-totals-to-digests' into fix-contributions-by-type
ChenglimEar Nov 30, 2023
70c9461
Merge branch 'fix-contributions-by-type' into upgrade-csvkit
ChenglimEar Dec 1, 2023
ea0229e
update digests to only show totals that we want to compare
ChenglimEar Dec 1, 2023
56834f0
add loans to total for contributions by type and origin
ChenglimEar Dec 6, 2023
599c72e
move totals logic out of main
ChenglimEar Dec 9, 2023
8d5684b
Merge branch 'master' into add-totals-to-digests
ChenglimEar Dec 9, 2023
55192e0
Merge branch 'add-totals-to-digests' into fix-contributions-by-type
ChenglimEar Dec 9, 2023
6faac9d
remove build directory for reset
ChenglimEar Dec 9, 2023
4573670
Merge branch 'add-totals-to-digests' into fix-contributions-by-type
ChenglimEar Dec 9, 2023
ee00671
add build directory back
ChenglimEar Dec 9, 2023
1db21ed
Merge branch 'add-totals-to-digests' into fix-contributions-by-type
ChenglimEar Dec 9, 2023
d13c972
Merge branch 'master' into add-totals-to-digests
ChenglimEar Dec 10, 2023
7b14a9a
re-run
ChenglimEar Dec 10, 2023
17a2f7c
Merge branch 'master' into fix-contributions-by-type
ChenglimEar Dec 10, 2023
788e22d
Merge branch 'fix-contributions-by-type' into upgrade-csvkit
ChenglimEar Dec 10, 2023
db58792
Merge branch 'add-totals-to-digests' into fix-contributions-by-type
ChenglimEar Dec 10, 2023
c1990cd
Merge branch 'master' into fix-contributions-by-type
ChenglimEar Dec 17, 2023
692b6ea
switch total expenditures calculator to use new candidate_summary view
ChenglimEar Dec 17, 2023
4bb1698
Merge branch 'fix-contributions-by-type' into upgrade-csvkit
ChenglimEar Dec 17, 2023
7d63c37
remove build directory
ChenglimEar Dec 17, 2023
1cfbab6
remove build directory
ChenglimEar Dec 17, 2023
075308e
Merge branch 'fix-contributions-by-type' into upgrade-csvkit
ChenglimEar Dec 17, 2023
463a32d
add generated build directory
ChenglimEar Dec 17, 2023
a3d11cb
Merge branch 'fix-contributions-by-type' into upgrade-csvkit
ChenglimEar Dec 17, 2023
40290cb
Merge branch 'master' into fix-contributions-by-type
ChenglimEar Dec 20, 2023
9664e2a
add report on candidate totals
ChenglimEar Dec 20, 2023
a0400b6
attempt to get python 3.9 to be used
ChenglimEar Dec 22, 2023
7113922
Merge branch 'master' into fix-contributions-by-type
ChenglimEar Dec 22, 2023
db38143
don't use sudo for pip install
ChenglimEar Dec 22, 2023
b763994
Merge branch 'fix-contributions-by-type' into upgrade-csvkit
ChenglimEar Dec 22, 2023
927d3de
remove build dir to reset
ChenglimEar Feb 3, 2024
a6fccb2
Merge branch 'master' into upgrade-csvkit
ChenglimEar Feb 3, 2024
43dd66c
Merge branch 'master' into upgrade-csvkit
ChenglimEar Feb 3, 2024
39423db
update build dir to match master as reset
ChenglimEar Feb 3, 2024
fbf2505
remove unused var in calculator
ChenglimEar Feb 3, 2024
731686c
match up calculator with master branch
ChenglimEar Feb 3, 2024
832b3db
update build dir to match master
ChenglimEar Feb 18, 2024
6eb44d1
Merge branch 'master' into upgrade-csvkit
ChenglimEar Feb 18, 2024
5806f1f
upgrade csvkit
ChenglimEar Feb 18, 2024
465b881
update build dir
ChenglimEar Feb 18, 2024
44d37b0
Merge branch 'upgrade-csvkit' into add-workflow-to-gen-data
ChenglimEar Feb 18, 2024
a5aec37
match schema to latest infered by old csvkit
ChenglimEar Feb 18, 2024
64aee65
make sure we are pushing to the same branch when deploying build
ChenglimEar Feb 18, 2024
f0c06e8
Merge branch 'master' into track-schema-changes
ChenglimEar Feb 18, 2024
50bd68e
specify the branch to push to for travis auto-deploy
ChenglimEar Feb 18, 2024
50ca568
Run `make clean download import process`
Feb 18, 2024
c9a3b76
Merge branch 'master' into track-schema-changes
ChenglimEar Feb 18, 2024
3abb21c
Run `make clean download import process`
Feb 18, 2024
ff45680
update build dir to match current
ChenglimEar Feb 18, 2024
c8b6f94
add schema.sql file
ChenglimEar Feb 18, 2024
8b18b65
Merge branch 'track-schema-changes' into upgrade-csvkit
ChenglimEar Feb 18, 2024
04c6375
don't deploy build on pull request build
ChenglimEar Feb 18, 2024
82cb37a
Merge branch 'track-schema-changes' into upgrade-csvkit
ChenglimEar Feb 18, 2024
8fb15e8
increase size of filer name for committees
ChenglimEar Feb 18, 2024
c9f4423
Run `make clean download import process`
Feb 18, 2024
8bba528
Merge branch 'master' into upgrade-csvkit
ChenglimEar Feb 20, 2024
46a5293
Run `make clean download import process`
Feb 20, 2024
6296668
clean up whitespace for some more candidate columns
ChenglimEar Feb 20, 2024
c9218b9
Run `make clean download import process`
Feb 20, 2024
6c06142
remove whitespace from referendums summary
ChenglimEar Feb 20, 2024
2e4c960
Merge branch 'upgrade-csvkit' of github.com:caciviclab/disclosure-bac…
ChenglimEar Feb 20, 2024
9e190ae
Run `make clean download import process`
Feb 20, 2024
678d721
remove commented out line
ChenglimEar Feb 27, 2024
93d718b
Merge branch 'upgrade-csvkit' of github.com:caciviclab/disclosure-bac…
ChenglimEar Feb 27, 2024
27881bf
remove build dir to reset
ChenglimEar Mar 2, 2024
17c6184
Merge branch 'master' into upgrade-csvkit
ChenglimEar Mar 2, 2024
750deaf
save new build dir
ChenglimEar Mar 2, 2024
e954c21
combine removal of leading and trailing white spaces into a single
ChenglimEar Mar 2, 2024
2c28ee3
Run `make clean download import process`
Mar 2, 2024
50bbec0
update build with recent fixes from main branch
ChenglimEar Mar 5, 2024
7efe43d
Merge branch 'upgrade-csvkit' of github.com:caciviclab/disclosure-bac…
ChenglimEar Mar 5, 2024
40ff1e8
re-use code to create table in bin/import-file
ChenglimEar Mar 5, 2024
b5954a2
remove build dir to reset
ChenglimEar Mar 5, 2024
93ec624
Run `make clean download import process`
Mar 5, 2024
9ec0ebd
Merge branch 'master' into upgrade-csvkit
ChenglimEar Mar 5, 2024
794d934
Merge branch 'upgrade-csvkit' of github.com:caciviclab/disclosure-bac…
ChenglimEar Mar 5, 2024
43db2e8
Run `make clean download import process`
Mar 5, 2024
e0ecb2b
clean up request to dump database schema
ChenglimEar Mar 9, 2024
d461b13
remove build dir to reset
ChenglimEar Mar 9, 2024
206f986
Merge branch 'master' into upgrade-csvkit
ChenglimEar Mar 9, 2024
08b6fc2
Run `make clean download import process`
Mar 9, 2024
12efe1d
remove build directory for refresh
ChenglimEar Mar 22, 2024
0a3da68
Merge branch 'upgrade-csvkit' of github.com:caciviclab/disclosure-bac…
ChenglimEar Mar 22, 2024
969df8a
Merge branch 'master' into upgrade-csvkit
ChenglimEar Mar 22, 2024
b86effd
Run `make clean download import process`
Mar 22, 2024
a3cc106
Merge branch 'master' into upgrade-csvkit
ChenglimEar Apr 8, 2024
06f0819
Run `make clean download import process`
Apr 8, 2024
e37d1f2
pick committee distinct on filer ID according to order of value in el…
ChenglimEar Apr 13, 2024
881733e
Merge branch '352-select-the-most-recent-committee-name' into upgrade…
ChenglimEar Apr 13, 2024
a558379
Run `make clean download import process`
Apr 13, 2024
0b160c9
Run `make clean download import process`
Apr 13, 2024
a043a76
remove check for Ballot_Measure_Election when looking for committee name
ChenglimEar Apr 13, 2024
a828a72
Merge branch '352-select-the-most-recent-committee-name' into upgrade…
ChenglimEar Apr 13, 2024
f0c00fa
Run `make clean download import process`
Apr 13, 2024
5006c28
force rebuild
ChenglimEar Apr 14, 2024
1f54189
Run `make clean download import process`
Apr 14, 2024
ffdc7e6
Merge branch 'upgrade-csvkit' into add-workflow-to-gen-data
ChenglimEar Apr 14, 2024
1d2b54d
wip: copying files over from queries repo
ckingbailey Oct 30, 2023
6f90e62
set data_dir to .local in tests
ckingbailey Oct 30, 2023
5769d8b
cp files from latest commit to ckingbailey v2 repo
ckingbailey Nov 1, 2023
9fbfef3
Run `make clean download import process`
May 25, 2024
1e99f65
Merge branch 'feat/pull-v2-api' into add-workflow-to-gen-data
ChenglimEar May 26, 2024
58a45bc
c/o stats from main
ckingbailey May 28, 2024
b984ebc
Run `make clean download import process`
May 28, 2024
e7a67f3
Run `make clean download import process`
May 29, 2024
8059e9f
change image used for workflow to generate website data to match vers…
ChenglimEar May 30, 2024
c1e48a5
Run `make clean download import process`
May 30, 2024
6a50288
set dev container and github actions to use the same postgres version
ChenglimEar May 30, 2024
8d65708
try action checkout v4
ChenglimEar May 30, 2024
59b34c2
print out some dir info to figure out why git thinks it is not a repo
ChenglimEar May 30, 2024
8adaf50
cause early git failure so we can try to fix it
ChenglimEar May 30, 2024
f961892
remove tab from github workflow file
ChenglimEar May 30, 2024
a79dce5
see if we can fix the git issue
ChenglimEar May 30, 2024
50055fb
remove test commands
ChenglimEar May 30, 2024
567c967
Run `make clean download import process`
May 30, 2024
87657f7
show version of key components when cleaning
ChenglimEar May 30, 2024
fecf2ec
Merge branch 'add-workflow-to-gen-data' of github.com:caciviclab/disc…
ChenglimEar May 30, 2024
ecc51fa
Run `make clean download import process`
May 30, 2024
4c3f104
add place to insert new downloads
ChenglimEar May 30, 2024
71695a9
Merge branch 'add-workflow-to-gen-data' of github.com:caciviclab/disc…
ChenglimEar May 30, 2024
5783cba
Feat: make D expenditures from NetFile V2 API (#334)
ckingbailey Jun 28, 2024
434ebde
Run `make clean download import process`
Jun 28, 2024
35fa76d
get image to be created with new branch and don't use the image during
ChenglimEar Jul 2, 2024
af8f51d
add explicit check for docker image in order to run jobs that require it
ChenglimEar Jul 2, 2024
bfd4b9a
log in to docker early
ChenglimEar Jul 2, 2024
3d8c799
build container if it's not there
ChenglimEar Jul 2, 2024
ef9e09a
try increasing size of filer name col
ChenglimEar Jul 2, 2024
6d4cad4
put shared postgres settings in global env vars
ChenglimEar Jul 3, 2024
e110b7e
clean up dev container
ChenglimEar Jul 3, 2024
0e6cca5
add post-create-command.sh back
ChenglimEar Jul 3, 2024
a6263c3
remove pwd in Dockerfile
ChenglimEar Jul 3, 2024
c83b157
Merge branch 'feat/pull-v2-api' into add-workflow-to-gen-data
ChenglimEar Jul 3, 2024
fb4e811
write csv from polars dataframe
ChenglimEar Jul 3, 2024
f929a6e
merge requirements for netfile v2 into main requirements file
ChenglimEar Jul 3, 2024
d7163d6
allow committee id to be null in H-Loan data
ChenglimEar Jul 3, 2024
3262819
Run `make clean download import process`
Jul 3, 2024
b067517
remove copy of download/requirements.txt from Dockerfile
ChenglimEar Jul 3, 2024
b756009
Run `make clean download import process`
Jul 3, 2024
8833de7
move new data to be imported to a different target in Makefile
ChenglimEar Jul 3, 2024
a1ff8f6
Merge branch 'add-workflow-to-gen-data' of github.com:caciviclab/disc…
ChenglimEar Jul 3, 2024
7986727
Run `make clean download import process`
Jul 3, 2024
5d7c25a
provide means to switch to ruby 2.7.1 if needed
ChenglimEar Jul 14, 2024
407b274
Merge branch 'add-workflow-to-gen-data' of github.com:caciviclab/disc…
ChenglimEar Jul 14, 2024
2760215
Run `make clean download import process`
Jul 14, 2024
e83eaa5
remove whitespace from data_warning column
ChenglimEar Jul 16, 2024
25c580c
Merge branch 'add-workflow-to-gen-data' of github.com:caciviclab/disc…
ChenglimEar Jul 16, 2024
f10a7a6
Run `make clean download import process`
Jul 16, 2024
a0a322f
make data_warning empty instead of null
ChenglimEar Jul 16, 2024
ca3d2a0
make make-null-empty executable
ChenglimEar Jul 16, 2024
fd68dd5
Run `make clean download import process`
Jul 16, 2024
9d0cb41
maintain a consistent order for the candidates report
ChenglimEar Jul 16, 2024
bcaafa8
Run `make clean download import process`
Jul 16, 2024
9759958
make null empty for data_warning in committees
ChenglimEar Jul 16, 2024
efa9973
Run `make clean download import process`
Jul 16, 2024
a6e89bd
output consistent order to ensure that output doesn't change when
ChenglimEar Jul 16, 2024
b4e4113
Run `make clean download import process`
Jul 16, 2024
710218c
set null committee name to empty string so that we can get consistent
ChenglimEar Jul 16, 2024
37eb448
Run `make clean download import process`
Jul 16, 2024
af67d7e
use floats everywhere when calculating totals in create-digests
ChenglimEar Jul 19, 2024
b46cccc
increase column size for instagram column in candidates table
ChenglimEar Jul 19, 2024
a07acd9
Run `make clean download import process`
Jul 19, 2024
2a538a3
add some additional totals for oakland-2024 election in digests.json to
ChenglimEar Jul 19, 2024
f2358af
Run `make clean download import process`
Jul 19, 2024
0d6d8a2
change election name in digests.json to include full date to correctly
ChenglimEar Jul 19, 2024
8da4ef6
Run `make clean download import process`
Jul 19, 2024
7ff8420
remove commented postgres 9.6 in workflow
ChenglimEar Jul 21, 2024
e0362c9
Run `make clean download import process`
Jul 21, 2024
f7d5769
only run netfile v2 download when credentials are set up
ChenglimEar Jul 21, 2024
d9c741b
Merge branch 'add-workflow-to-gen-data' of github.com:caciviclab/disc…
ChenglimEar Jul 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
increase column size for instagram column in candidates table
  • Loading branch information
ChenglimEar committed Jul 19, 2024
commit b46cccc7690c7e7b2e508eba2e29d880a09e7af9
2 changes: 1 addition & 1 deletion dbschema/candidates.sql
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@ CREATE TABLE candidates (
"Website" VARCHAR(68),
"Twitter" VARCHAR(17),
"Facebook" VARCHAR(91),
"Instagram" VARCHAR(23),
"Instagram" VARCHAR(91),
"Party_Affiliation" VARCHAR(11),
"Occupation" VARCHAR(78),
"Bio" VARCHAR(1295),
Loading