chore/merge upstream 2024071101

[onimsha]

• [minor] proper assistant line length
• hide popup for non admin + if search is disabled
• finished first draft
• Transform HTML links to markdown behind config option (Transform HTML links to markdown behind config option onyx-dot-app/onyx#1671)
• combined the input fields
• Update handle_message.py
• google drive ignores shortcut filetypes now
• Improve persona access
• Fix image generation output
• Enable Internet Search for Deployment Options (Enable Internet Search for Deployment Options onyx-dot-app/onyx#1684)
• Add functional thread modification endpoints (add functional thread modification endpoints onyx-dot-app/onyx#1668)
• Discourse Edge Case (Discourse Edge Case onyx-dot-app/onyx#1685)
• Add migration for usage reports
• Add script for forced connector deletion (add forced connector deletion onyx-dot-app/onyx#1683)
• Relari (Relari onyx-dot-app/onyx#1687)
• modify script deletion name (modify script deletion name onyx-dot-app/onyx#1690)
• add basic 403 support for healthcheck (add basic 403 support for healthcheck onyx-dot-app/onyx#1689)
• fixed confluence breaking on unknown filetypes (fixed confluence breaking on unknown filetypes onyx-dot-app/onyx#1698)
• Touchup (Touchup onyx-dot-app/onyx#1702)
• Confluence: add pages labels indexation (Confluence: add pages labels indexation onyx-dot-app/onyx#1635)
• Initial EE features (Adding required permission for the jobs #3)
• Add User Groups (a.k.a. RBAC) (Fix the wrong parameter on the model server build workflow #4)
• Move user group syncing to Celery Beat
• Update ee supervisord
• Admin Analytics/Query History dashboards (build arm64 images #6)
• Enable EE features for no-letsencrypt deployment (chore/debugging celery #9)
• Update default SAML config location (Disable sslmode #10)
• Fix group prefix (Use proper sqlalchemy connection arguments #11)
• Fix Rebase Issue
• Danswer EE Version Text (Use a different config args for aysnc db #12)
• Fix EE import
• Improvements to Query History (Disable autoscale in celery #17)
• Fix missing supervisord change from Danswer MIT (Disable sqlalchemy connection pooling #18)
• Add new env variable to EE supervisord
• Mark indexing jobs as ee when running ee supervisord
• Fix sidebar typo
• Fix frontend typing rebase issue
• Fix analytics + query history
• Re-style user group pages
• Fix /chat-session-history/{chat_session_id} endpoint when auth is enabled
• Make query history fetch client-side
• Fix viewing other users' chat histories in query history
• CVEs (main upstream #26)
• Add API key generation in the UI + allow it to be used across all endpoints
• Introduce EE only Backend APIs (Apply re-branding modifications #29)
• Fix users page with API keys + add spinner on key creation
• Add global prefix to EE endpoints
• Fix limit/offset for document-search endpoint
• Embedding Model Swap Changes (Update built-in assistant #35)
• Make public/not-public selector prettier
• Basic Chat API (chore/merge upstream 2024050601 #38)
• Fix Empty Chat for API (chore/merge upstream 2024061401 #39)
• Simple API No Longer Require Specify Prompt (Use a larger model for reranking #40)
• One Shot API No Stream (chore/merge upstream 2024061701 #41)
• Fix user group deletion bug
• Add env variable to control hash rounds
• Fix small answer-with-quote bug
• Change runner for backend
• Metadata filter for document search API (remove some models to save memory during build #53)
• Private personas doc sets (Set static variable in the workflow #52)
• Fix SAML for /manage/me
• Rebase search changes to EE APIs (chore/merge upstream 2024091301 #55)
• Fix Query History (matching code with latest upstream #56)
• Formatter (Restore Standard Answer feature back #57)
• Query History Now Handles Old Messages (Disable pre-downloading models #58)
• Include User in Query Export (Fix conflicts #59)
• Allow Optional Rerank in APIs (Hack: Support prefixed channel config #60)
• Whitelableing for Logo / Name via Admin panel
• Add custom-styling ability via themes
• Small fixes + adding 'Powered by Danswer'
• Custom banner
• Add names to API Keys (Adding support for AT connector #63)
• Add custom analytics script
• FE build fix
• Make EE containers public changes (Adding support for VertexAI #67)
• Add Chunk Context options for EE APIs (Switch base image to run on GPU Node #68)
• Add message id to simple message endpoint (Bumping deps #69)
• Add automated auth checks for ee
• Permission Sync Framework (Temporarily disable user whitelist check #44)
• Add back ChatBanner
• Fix ee redirect
• Fix rebase issue with public assistants
• Better Naming for API Keys (Making a few UI tweaks #76)
• Allow Empty API Names (Return None if field is missing in Airtable record #77)
• Remove Nested Session (Use bigger favicon, and update some texts #78)
• Reduce sync jobs batch size (chore/merge upstream 2024112601 #79)
• Query History Include Feedback Text (Add back GTM tracking #80)
• Fix Web Compile Issue (chore/merge upstream 2024112801 #81)
• Token Rate Limiting
• Fix rebase issue with UI-based LLM selection
• Remove unused Chat.tsx file
• Fix rebase issue with file_store
• Fix /send-message-simple-api endpoint
• Track Slack questions Autoresolved (Add Filters to UI onyx-dot-app/onyx#86)
• Change secret name
• Fix document lock acquisition for user group sync
• Enable Encryption
• Fix rebase issue with dev compose file
• Add pagination to user group syncing
• Fix mypy
• Don't get duplicate docs during user group syncing
• Move web build to a matrix build
• Fix Rebase Issues (DAN-116 Google Drive Poll connector onyx-dot-app/onyx#92)
• Fix formatting issues (File connector onyx-dot-app/onyx#93)
• Fix rebase issue
• Add back custom logo/name to sidebar header
• Fix user groups issue from rebase
• Add whitelabeled name to login page
• Enable Dedup Flag for Doc Search Endpoint
• Fix rebase issue
• Allow basic seeding of Danswer via env variable
• Try different runner groups for each build
• Fix backend build
• Add support for private file connectors
• Query history speed fix (DAN-139 web connector recursion onyx-dot-app/onyx#109)
• Minor/ee optional branding (Fix missing sources when auth is on onyx-dot-app/onyx#105)
• stronger wording
• Add persistent name and logo seeding (Fix connection pooling onyx-dot-app/onyx#107)
• minor build fix
• formatting
• Fix persona access for answer-with-quote API
• Add empty assets folder
• Fix seed config when left empty
• brought out the UsersResponse interface (Update semver version onyx-dot-app/onyx#119)
• Usage reports (images can't be displayed properly onyx-dot-app/onyx#118)
• Prepare EE to merge with MIT
• Change EE environment variable name
• Ignore messages from Slack's official bot (Ignore messages from Slack's official bot onyx-dot-app/onyx#1703)
• Developer Env Setup (Developer Environment Setup onyx-dot-app/onyx#1710)
• Bugfix for Selected Doc when the message it is selected from failed (Bugfix for Selected Doc when the message it is selected from failed onyx-dot-app/onyx#1711)
• Catch LLM Generation Failure (Catch LLM Generation Failure onyx-dot-app/onyx#1712)
• Minor Update to UI (Minor Update to UI onyx-dot-app/onyx#1692)
• Small template fix
• Make seeded model default
• add copy-paste images (add copy-paste images onyx-dot-app/onyx#1722)
• Fix search tool
• Touchup for Multilingual Users (Touchup for Multilingual Users onyx-dot-app/onyx#1725)
• Chat overflow (Chat overflow onyx-dot-app/onyx#1723)
• Fix share for insecure:
• Use FiEdit2 as the standard edit icon
• Add more logging for num workers in simple job client
• add workflows that automate docker builds against merge group events
• Tidy up Actions ported from EE (Tidy up Actions ported from EE onyx-dot-app/onyx#1732)
• Feature/scroll (Feature/scroll onyx-dot-app/onyx#1694)
• Remove duplicate tool check
• Blob Storage (Blob Storage onyx-dot-app/onyx#1705)
• Fixes DAN-189 (safari bug in admin). Removed td/absolute positioning behavior which is unde… (Fixes DAN-189 (safari bug in admin). Removed td/absolute positioning behavior which is unde… onyx-dot-app/onyx#1718)
• remove obsolete comment
• Small readme improvement (Small readme improvement onyx-dot-app/onyx#1735)
• Pass headers into image generation (Pass headers into image generation onyx-dot-app/onyx#1739)
• add boto3 typing to default requirements (add boto3 typing to default requirements onyx-dot-app/onyx#1740)
• Added TTL to EE Celery tasks (Added TTL to EE Celery tasks onyx-dot-app/onyx#1713)
• renamed alembic table (fixed alembic upgrade onyx-dot-app/onyx#1741)
• Disallowed simultaneous pruning jobs (Disallowed simultaneous pruning jobs onyx-dot-app/onyx#1704)
• Feature/assistants (Feature/assistants onyx-dot-app/onyx#1581)
• Fix fast models
• Bugfix/scroll (Bugfix/scroll onyx-dot-app/onyx#1748)
• Fix slackbot citation images
• Fix docker build
• Fix misc chat bugs
• added settings and persona seeding options (added settings and persona seeding options onyx-dot-app/onyx#1742)
• Fix build caused by bad seeding config
• Fix "cannot access local variable" for bot direct messages (Fix "cannot access local variable" for bot direct messages onyx-dot-app/onyx#1737)
• Notion date filter fix (Notion date filter fix onyx-dot-app/onyx#1755)
• Add litellm.set_verbose support
• Pass through API base to ImageGenerationTool
• allowed arbitrary types to handle the sqlalchemy datatype (allowed arbitrary types to handle the sqlalchemy datatype onyx-dot-app/onyx#1758)
• Update Assistants Creation UI (Update Assistants Creation UI onyx-dot-app/onyx#1714)
• minor updates to assistant UI (Minor updates to the new assistants page onyx-dot-app/onyx#1771)
• Add UI support for github configs
• fixed salesforce url generation (updated salesforce url generation onyx-dot-app/onyx#1777)
• Fix shared chats
• Added search quality testing pipeline (Added search quality testing pipeline onyx-dot-app/onyx#1774)
• Docker Compose Eval Pipeline Cleanup (Docker Compose Eval Pipeline Cleanup onyx-dot-app/onyx#1780)
• New Env File for Eval (New Env File for Eval onyx-dot-app/onyx#1781)
• Ignore Eval Files (Ignore Eval Files onyx-dot-app/onyx#1782)
• Evaluation Pipeline Touchup (Evaluation Pipeline Touchup onyx-dot-app/onyx#1783)
• Eval Script Incremental Write (Eval Script Incremental Write onyx-dot-app/onyx#1784)
• Standard Answers (Standard Answers onyx-dot-app/onyx#1753)
• Internet Search Tool (Internet Search onyx-dot-app/onyx#1666)
• fix gpt-4o image issue (fix gpt-4o image issue onyx-dot-app/onyx#1786)
• fix wrapping in error hover connector (fix wrapping in error hover connector onyx-dot-app/onyx#1790)
• Info propagating to allow Chunk Merging (Info propagating to allow Chunk Merging onyx-dot-app/onyx#1794)
• Break the Danswer LLM logging from LiteLLM Verbose (Break the Danswer LLM logging from LiteLLM Verbose onyx-dot-app/onyx#1795)
• add basic table wrapping (add basic table wrapping onyx-dot-app/onyx#1791)
• Fix citations + unit tests (Fix citations + unit tests onyx-dot-app/onyx#1760)
• Confluence Connector to Sync Child pages only (Confluence Connector to Sync Child pages only onyx-dot-app/onyx#1629)
• Add content to /document-search response
• slackbot doesnt respond without citations/quotes (slackbot doesnt respond without citations/quotes onyx-dot-app/onyx#1798)
• Merge Sections Logic (Merge Sections Logic onyx-dot-app/onyx#1801)