Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emnlp #7

Open
wants to merge 93 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
a2ca02a
Add ATP-specific generate DVs workflow
vinhowe May 12, 2023
095912c
Apparently we could generate DVs automatically
vinhowe May 12, 2023
298d028
Add survey cost estimation code
vinhowe May 16, 2023
b6a536e
Add code to automatically generate ATP configs
vinhowe May 16, 2023
a65478c
Lowercase variables because it works for neurips
vinhowe May 16, 2023
39026e1
Add 'culling sampled below n' code (note!)
vinhowe May 16, 2023
f937f59
Merge branch 'main' into neurips
vinhowe May 19, 2023
80110d9
Add output for debugging prompt construction
vinhowe May 19, 2023
08e9bb5
Revert "Lowercase variables because it works for neurips"
vinhowe May 19, 2023
7e7adef
Undo ValidOption lowercasing
vinhowe May 19, 2023
7435097
Revert "Add 'culling sampled below n' code (note!)"
vinhowe May 20, 2023
e0df35c
Remove n_culle_sampled_below everywhere
vinhowe May 20, 2023
97e84c8
Unsort imports in survey.py
vinhowe May 20, 2023
b013cb9
Ignore .DS_Store
vinhowe May 20, 2023
298cf23
Merge branch 'main' into emnlp
vinhowe May 20, 2023
a990667
s/config_filename/variables_filename/g
vinhowe May 20, 2023
5fc9c00
Filter out out-of-schema variable values
vinhowe May 20, 2023
bd7a057
Working async openai sampler impl
vinhowe May 20, 2023
490e07c
Reformat
vinhowe May 20, 2023
81c22f7
Remove extra comment
vinhowe May 20, 2023
3bd43df
Update example_configure_survey.py
vinhowe May 20, 2023
7b68fa6
Remove unused argparse import
vinhowe May 20, 2023
fea3a1f
Reformat
vinhowe May 20, 2023
b4a8fe9
Merge branch 'emnlp' into async-openai-sampling
vinhowe May 20, 2023
3728470
Remove unused prompt printing
vinhowe May 20, 2023
a218599
Merge branch 'emnlp' into async-openai-sampling
vinhowe May 20, 2023
2877aab
Remove comments for copilot
vinhowe May 20, 2023
4e8155d
Merge branch 'emnlp' into async-openai-sampling
vinhowe May 20, 2023
10b50a5
replace responses.csv -> data.csv
vinhowe May 20, 2023
a41a1bd
Update folder structure.
alexgshaw May 22, 2023
27615c8
Add style check GitHub workflow
vinhowe May 22, 2023
056761b
Merge branch 'emnlp' into async-openai-sampling
vinhowe May 22, 2023
8035a00
Bug fix + add slot for response object.
alexgshaw May 22, 2023
98b4acf
Fix index typing error.
alexgshaw May 22, 2023
687dcd0
Merge branch 'emnlp' into async-openai-sampling
vinhowe May 22, 2023
bd381ba
Merge branch 'emnlp' into async-openai-sampling
vinhowe May 22, 2023
a7febdf
Update async openai sampler to return response
vinhowe May 22, 2023
82c356c
Add ordinal property to variables.json
vinhowe May 22, 2023
08f02c5
Handle AutoModel async
vinhowe May 22, 2023
05ddeef
Merge pull request #8 from BYU-PCCL/async-openai-sampling
vinhowe May 22, 2023
2ed52ae
Merge branch 'emnlp' into add-ordinal-to-variables
vinhowe May 22, 2023
2c31a63
Merge pull request #11 from BYU-PCCL/add-ordinal-to-variables
alexgshaw May 22, 2023
64afba1
Fix survey sampling.
alexgshaw May 22, 2023
23bb834
Merge branch 'emnlp' of https://github.com/BYU-PCCL/lm-survey into emnlp
alexgshaw May 22, 2023
2b82b88
Update atp configuration
vinhowe May 22, 2023
f126bc5
Add ordinal functionality to Question class.
alexgshaw May 22, 2023
639d687
Merge branch 'emnlp' of https://github.com/BYU-PCCL/lm-survey into emnlp
alexgshaw May 22, 2023
3740cfc
Rework folder structure.
alexgshaw May 22, 2023
d7198fb
Temp. revert "Add style check GitHub workflow"
vinhowe May 23, 2023
b0b14e5
bug fix.
alexgshaw May 23, 2023
aee5f3c
Minor bug fix on question.
alexgshaw May 23, 2023
f260e71
rename schema to variable
alexgshaw May 23, 2023
83bdc97
gitignore update
alexgshaw May 23, 2023
ea66dd5
Added crude representativeness scoring
chrisrytting May 24, 2023
8008a9f
Merge branch 'emnlp' of github.com:BYU-PCCL/lm-survey into emnlp
chrisrytting May 24, 2023
ac835b2
Update estimate survey to experiment config file
vinhowe May 24, 2023
d47640a
Merge branch 'emnlp' of github.com:BYU-PCCL/lm-survey into emnlp
vinhowe May 24, 2023
e3c4c8e
breadth experiment
chrisrytting May 24, 2023
f9347ac
Merge branch 'emnlp' of github.com:BYU-PCCL/lm-survey into emnlp
chrisrytting May 24, 2023
67bbf30
Merge branch 'emnlp' of github.com:BYU-PCCL/lm-survey into emnlp
vinhowe May 24, 2023
5940ff0
Add create atp experiment script
vinhowe May 24, 2023
39c2fe4
Check in variables
vinhowe May 24, 2023
95da00f
Clean up create atp experiment imports
vinhowe May 24, 2023
562fb9d
Do formatting for create atp experiment
vinhowe May 24, 2023
28e3ac9
Update check survey prompts for experiments
vinhowe May 24, 2023
4e79da7
Update estimate survey to experiment config file
vinhowe May 24, 2023
cde6e24
Fix ATP configuration script
vinhowe May 24, 2023
9076e10
Use ordinals to find invalid options
vinhowe May 24, 2023
9eba1c9
Fix
vinhowe May 24, 2023
f0cf51d
Fix rate limit error import
vinhowe May 25, 2023
f510c3e
Bump up rate limit to what OpenAI says we have
vinhowe May 25, 2023
9aaead3
Push updates variables
vinhowe May 26, 2023
ac8e3fb
Add force flag.
alexgshaw May 26, 2023
6e9d77f
Sampler fix.
alexgshaw May 26, 2023
0a24acd
Added logging, added functionality to fill in missing response_objects
chrisrytting May 26, 2023
ad4525d
Merge branch 'emnlp' of github.com:BYU-PCCL/lm-survey into emnlp
chrisrytting May 26, 2023
b058a5f
Bug fix.
alexgshaw May 26, 2023
34115df
Merge branch 'emnlp' of github.com:BYU-PCCL/lm-survey into emnlp
chrisrytting May 27, 2023
5b3c884
added some helpers
chrisrytting May 29, 2023
07fdbbe
Added tests for infilling
chrisrytting May 29, 2023
21b8e75
Added infilling ability, some more logging, and some extra DVS functi…
chrisrytting May 29, 2023
446672c
Removed data push mistake
chrisrytting May 29, 2023
835da77
Merge branch 'emnlp' of github.com:BYU-PCCL/lm-survey into emnlp
vinhowe May 29, 2023
1aff991
Remove unused import in async sampler
vinhowe May 29, 2023
8f7e6ad
Add updated estimate_survey.py
vinhowe May 30, 2023
849e91e
Minor fixes.
alexgshaw May 31, 2023
91b8041
Get rid of horrible rate limit print
vinhowe May 31, 2023
e1b1c52
Merge branch 'emnlp' of github.com:BYU-PCCL/lm-survey into emnlp
vinhowe May 31, 2023
f65c797
Made the rep calculation neater
chrisrytting Jun 2, 2023
c96a889
Merge branch 'emnlp' of github.com:BYU-PCCL/lm-survey into emnlp
chrisrytting Jun 2, 2023
0909387
Merge branch 'emnlp' of github.com:BYU-PCCL/lm-survey into emnlp
vinhowe Jun 2, 2023
df555c6
Added weighting and an ability to extract D_H from opinionqa dataset
chrisrytting Jun 3, 2023
a0ba33e
Merge branch 'emnlp' of github.com:BYU-PCCL/lm-survey into emnlp
chrisrytting Jun 3, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Unsort imports in survey.py
vinhowe committed May 20, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 97e84c83d80f40d2465f7b2c098f98887bdbd7c4
8 changes: 4 additions & 4 deletions lm_survey/survey/survey.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
import argparse
import functools
import json
import os
import typing
from pathlib import Path
@@ -9,13 +6,16 @@
import pandas as pd
from sklearn.metrics import normalized_mutual_info_score

from lm_survey.prompt_templates import INDEPENDENT_VARIABLE_SUMMARY_TEMPLATE
from lm_survey.survey.dependent_variable_sample import (
Completion,
DependentVariableSample,
)
from lm_survey.survey.question import Question, ValidOption
from lm_survey.survey.variable import Variable
from lm_survey.prompt_templates import INDEPENDENT_VARIABLE_SUMMARY_TEMPLATE
import json
import functools
import argparse


class Survey: