AIChat full-page storage #25876

petemill · 2024-10-08T07:02:12Z

Adds AIChatDatabase class which purely looks at persisting and retrieving conversation and conversation entry items.
Modified AIChatService to manage data going in and out of the database
Adds relevant uuid fields and modifies relevant constructors and unit tests do deal with them

TODO:

iOS isn't compiling due to not supporting optional int "ID" field for SiteInfo struct. I'll wrap this with another optional struct or convert to string.
For browser data deletion, when "browsing history" is selected, also delete AI Chat conversations which were created or modified within the specified timeframe and have associated content.
Implement a browsing_data::BrowsingDataCounter to show how many conversations will be deleted, possibly how many megabytes total the DB is (follow-up, nice to have).

brave-builds · 2024-10-09T18:16:09Z

A Storybook has been deployed to preview UI for the latest push

fallaciousreasoning

Wow, this is looking great @petemill! Lets hold off merging this till #25800 lands so we don't have to get the code reviewed twice.

components/ai_chat/core/common/mojom/ai_chat.mojom

fallaciousreasoning · 2024-10-15T00:24:09Z

browser/browsing_data/brave_browsing_data_remover_delegate.cc

-  // It is prepared for future implementation.
+  // Delete all Leo data
+  ai_chat::AIChatServiceFactory::GetForBrowserContext(profile_)
+      ->ClearAllHistory();


do we need to do a null check here? GetForBrowserContext can return null.

Maybe a CHECK if we expect it to never happen, for better crash logs.

fallaciousreasoning · 2024-10-15T00:25:46Z

browser/ui/webui/ai_chat/ai_chat_ui_page_handler.cc

@@ -6,7 +6,9 @@
 #include "brave/browser/ui/webui/ai_chat/ai_chat_ui_page_handler.h"

 #include <memory>
+#include <string>


nit: either revert the file, or move the <string> include into the header file (as it depends on it too)

fallaciousreasoning · 2024-10-15T00:27:41Z

browser/ui/webui/settings/brave_settings_leo_assistant_handler.cc

-  profile_->GetPrefs()->ClearPref(ai_chat::prefs::kLastAcceptedDisclaimer);
-  g_brave_browser_process->process_misc_metrics()
-      ->ai_chat_metrics()
-      ->RecordReset();


Note for other reviewers: This call still happens in ClearAllHistory

fallaciousreasoning · 2024-10-15T00:30:04Z

components/ai_chat/core/browser/ai_chat_database.cc

+#include "base/check.h"
+#include "base/strings/string_split.h"
+#include "base/time/time.h"
+#include "brave/components/ai_chat/core/common/mojom/ai_chat.mojom-shared.h"


nit: remove

components/ai_chat/core/browser/conversation_handler.cc

fallaciousreasoning · 2024-10-15T00:56:31Z

components/ai_chat/core/browser/conversation_handler.cc

@@ -873,10 +933,13 @@ void ConversationHandler::AddToConversationHistory(
    return;
  }

+  if (!turn->uuid.has_value()) {
+    turn->uuid = base::Uuid::GenerateRandomV4().AsLowercaseString();


optional: could be worth a NewId function in the anonymous namespace as we call this a few times. Pretty simple though so up to you

components/ai_chat/core/browser/conversation_handler.cc

fallaciousreasoning · 2024-10-15T01:00:04Z

components/ai_chat/core/browser/ai_chat_service.cc

+    ai_chat_db_.Reset();
+    return;
+  }
+  DVLOG(0) << "AIChat DB initialized";


DVLOG(0)? this doesn't seem that different in importance to the other logs

components/ai_chat/core/browser/ai_chat_service.h

darkdh · 2024-10-21T17:44:38Z

components/ai_chat/core/browser/ai_chat_database.h

+  bool AddConversationEntry(
+      std::string_view conversation_uuid,
+      mojom::ConversationTurnPtr entry,
+      std::optional<std::string> editing_id = std::nullopt);


This is already the default value

darkdh · 2024-10-21T18:25:51Z

components/ai_chat/core/browser/ai_chat_service.cc

+  ai_chat_db_.AsyncCall(&AIChatDatabase::IsInitialized)
+      .Then(base::BindOnce(&AIChatService::OnDBInit,
+                           weak_ptr_factory_.GetWeakPtr()));
+}
+
+void AIChatService::OnDBInit(bool success) {
+  CHECK(ai_chat_db_);
+  if (!success) {
+    LOG(ERROR) << "Failed to initialize AIChat DB";
+    // Ensure any pending tasks are cancelled. Sometimes, especially during unit
+    // tests that are very quick to run, initilization might fail because of
+    // shutdown but the next task in the sequence still executes, causing a
+    // crash because of bad database state.
+    ai_chat_db_.Reset();
+    return;
+  }


I think it is better we do

ai_chat_db_.AsyncCall(&AIChatDatabase::Init) .WithArgs(storage_dir.Append(kDBFileName)) .Then(base::BindOnce(&AIChatService::OnDBInit, weak_ptr_factory_.GetWeakPtr())); }

instead of doing it in constructor to eliminate the unnecessary intermediate state is_initialized_ which cause us to possibly check initialized in the middle of initializing process.

I did start out by having a separate public AIChatDatabase::Init function that consumers would call, but it seemed redundant because what's the point of constructing the class without Init being run, and also I didn't want to have to check that it was init'ed on every other function call.

I could change it back, but just thinking about the reasoning - is it really possible for is_initialized_ to be checked in the middle of the initializing process when all the other functions will be called on the sequence and I assume that guarantees Init will be finished, since it's called in the constructor? I know I'm probably wrong about this so please let me know!

Maybe I read your comments in a wrong way that makes me think that is possible to interrupt it. But it seems impossible due to the same reason you mentioned, constructing is posted to the sequenced task runner we chose and then when we call other functions, it will be posted to the same sequence as next task.

But even with current code, we still rely on consumer to check if it is successfully inited and it is not safe to assume that it would successfully initialized upon calling other public functions. So we still miss checking is_initialized_ in all public functions because init might fail.

darkdh · 2024-10-21T18:34:44Z

components/ai_chat/core/browser/ai_chat_database.cc

+}
+
+bool AIChatDatabase::CreateConversationTable() {
+  static const std::string kCreateConversationTableQuery =


static constexpr char kCreateConversationTableQuery[] and should apply to all the static queries

darkdh · 2024-10-21T18:56:39Z

components/ai_chat/core/browser/ai_chat_database.cc

+}
+
+std::optional<std::string> AIChatDatabase::DecryptOptionalColumnToString(
+    sql::Statement& statement,


This should be const sql::Statement&

darkdh · 2024-10-21T18:57:44Z

components/ai_chat/core/browser/ai_chat_database.cc

+  auto column_type = statement.GetColumnType(index);
+  // Don't allow non-BLOB types
+  if (column_type != sql::ColumnType::kBlob) {


if (statement.GetColumnType(index) != sql::ColumnType::kBlob) {

This is a style nit, right?

yes, it doesn't need named variable for one time use unless it is something like callback that would increase level of indention which makes it harder to read.

darkdh · 2024-10-21T20:50:22Z

components/ai_chat/core/browser/ai_chat_service.cc

+        std::nullopt, std::nullopt, 0, false, false);
+    mojom::ConversationPtr conversation =
+        mojom::Conversation::New(conversation_uuid, "", base::Time::Now(),
+                                 false, std::move(non_content));


inline mojom::SiteInfo::New

darkdh · 2024-10-21T21:04:36Z

components/ai_chat/core/browser/ai_chat_service.cc

+    std::move(callback).Run(nullptr);
+    return;
+  }
+  CHECK(ai_chat_db_);


darkdh · 2024-10-21T21:18:47Z

components/ai_chat/core/browser/ai_chat_service.cc

+           << " with data: " << data->entries.size() << " entries and "
+           << data->associated_content.size() << " contents";
+  mojom::Conversation* conversation =
+      conversations_.at(conversation_uuid.data()).get();


It will crash when the conversation is no longer in the map. We can't guarantee that won't happen between AIChatDatabase::GetConversationData and getting the result back

components/ai_chat/core/browser/ai_chat_database.cc

components/ai_chat/core/browser/conversation_handler.cc

browser/resources/settings/brave_overrides/clear_browsing_data_dialog.ts

Delete all AI Chat history from settings page

github-actions · 2024-10-30T18:30:46Z

[puLL-Merge] - brave/brave-core@25876

Description

This PR implements persistent storage for AI Chat conversations using a new AIChatDatabase class. It modifies the AIChatService and ConversationHandler to use this database for storing and retrieving conversation data. The changes also include updates to the mojom definitions, test utilities, and various unit tests to support the new functionality.

Changes

Changes

components/ai_chat/core/browser/ai_chat_database.cc and .h:
- New files implementing the AIChatDatabase class for persistent storage of conversations.
components/ai_chat/core/browser/ai_chat_service.cc and .h:
- Modified to use the new AIChatDatabase for storing and retrieving conversations.
- Added methods for database initialization and conversation data management.
components/ai_chat/core/browser/conversation_handler.cc and .h:
- Updated to support persistent storage of conversation entries.
- Modified event handling for conversation changes.
components/ai_chat/core/common/mojom/ai_chat.mojom:
- Updated definitions for Conversation, ConversationTurn, and related structures to support persistent storage.
components/ai_chat/core/browser/test_utils.cc and .h:
- New files with utility functions for testing conversation-related functionality.
Various unit test files:
- Updated to account for the new persistent storage functionality.
browser/ai_chat/ai_chat_service_factory.cc:
- Modified to include the new OSCryptAsync and profile path parameters in AIChatService construction.

Possible Issues

The PR introduces a significant amount of new code, which may increase the complexity of the AI Chat system and potentially introduce new bugs.
The changes to the database schema and conversation structure may require careful migration handling for existing users.

Security Hotspots

components/ai_chat/core/browser/ai_chat_database.cc:
- The database operations involve encryption and decryption of sensitive data. Ensure that the encryption methods are properly implemented and that no unencrypted data is accidentally stored or transmitted.
components/ai_chat/core/browser/ai_chat_service.cc:
- The ClearAllHistory method deletes all conversation data. Ensure that this operation is properly secured and can only be triggered by authorized users or processes.
components/ai_chat/core/browser/conversation_handler.cc:
- The SetArchiveContent method stores content from associated web pages. Ensure that this content is properly sanitized and does not introduce XSS or other injection vulnerabilities.

petemill self-assigned this Oct 8, 2024

github-actions bot added the CI/storybook-url Deploy storybook and provide a unique URL for each build label Oct 8, 2024

petemill changed the base branch from master to aichat-fullpage October 8, 2024 07:02

petemill force-pushed the aichat-fullpage-storage branch 3 times, most recently from d881677 to ff534cf Compare October 9, 2024 07:12

petemill force-pushed the aichat-fullpage-storage branch from b5cc7ce to 7f62f91 Compare October 10, 2024 01:53

petemill added the needs-security-review label Oct 10, 2024

petemill force-pushed the aichat-fullpage-storage branch 8 times, most recently from b6bf7d6 to 560779c Compare October 12, 2024 05:08

fallaciousreasoning force-pushed the aichat-fullpage branch from 616868a to c2d2026 Compare October 13, 2024 22:39

petemill force-pushed the aichat-fullpage-storage branch 3 times, most recently from bb6686b to 85e2bed Compare October 14, 2024 22:36

petemill marked this pull request as ready for review October 14, 2024 22:38

petemill requested review from a team as code owners October 14, 2024 22:38

petemill requested review from yrliou and darkdh October 14, 2024 22:40

petemill force-pushed the aichat-fullpage-storage branch from 85e2bed to b1fe7ad Compare October 14, 2024 22:44

petemill requested a review from fallaciousreasoning October 14, 2024 23:04

petemill force-pushed the aichat-fullpage-storage branch from b1fe7ad to cdccd62 Compare October 15, 2024 00:00

fallaciousreasoning reviewed Oct 15, 2024

View reviewed changes

diracdeltas requested review from thypon and ShivanKaul October 16, 2024 16:46

darkdh reviewed Oct 22, 2024

View reviewed changes

petemill force-pushed the aichat-fullpage branch from 13d9fbf to 0101ade Compare October 22, 2024 17:36

petemill requested a review from a team as a code owner October 22, 2024 17:36

petemill force-pushed the aichat-fullpage branch from 0101ade to 988abbe Compare October 22, 2024 18:12

petemill force-pushed the aichat-fullpage-storage branch 2 times, most recently from e24b13a to 5c43459 Compare October 22, 2024 18:45

fallaciousreasoning reviewed Oct 23, 2024

View reviewed changes

browser/resources/settings/brave_overrides/clear_browsing_data_dialog.ts Show resolved Hide resolved

petemill force-pushed the aichat-fullpage branch from dca362b to df33be5 Compare October 25, 2024 17:13

petemill force-pushed the aichat-fullpage-storage branch from 7d10150 to 82a0b3d Compare October 29, 2024 05:29

Base automatically changed from aichat-fullpage to master October 30, 2024 16:19

nullhook and others added 4 commits October 30, 2024 11:29

aichat: added disk storage

2207ed5

Persist all conversation and conversation entry data

1559840

Delete all AI Chat history from settings page

Clean up AI Chat settings browser data deleter strings

6c5c036

conversation metrics simplified

6677620

petemill force-pushed the aichat-fullpage-storage branch from 82a0b3d to 6677620 Compare October 30, 2024 18:30

github-actions bot added the puLL-Merge label Oct 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AIChat full-page storage #25876

AIChat full-page storage #25876

petemill commented Oct 8, 2024 •

edited

Loading

brave-builds commented Oct 9, 2024

fallaciousreasoning left a comment •

edited

Loading

fallaciousreasoning Oct 15, 2024

fallaciousreasoning Oct 15, 2024

fallaciousreasoning Oct 15, 2024

fallaciousreasoning Oct 15, 2024

fallaciousreasoning Oct 15, 2024

fallaciousreasoning Oct 15, 2024

darkdh Oct 21, 2024

darkdh Oct 21, 2024

petemill Oct 22, 2024

darkdh Oct 22, 2024

darkdh Oct 22, 2024

darkdh Oct 21, 2024

darkdh Oct 21, 2024

darkdh Oct 21, 2024

petemill Oct 22, 2024

darkdh Oct 22, 2024

darkdh Oct 21, 2024

darkdh Oct 21, 2024

darkdh Oct 21, 2024

github-actions bot commented Oct 30, 2024

Changes

AIChat full-page storage #25876

Are you sure you want to change the base?

AIChat full-page storage #25876

Conversation

petemill commented Oct 8, 2024 • edited Loading

brave-builds commented Oct 9, 2024

fallaciousreasoning left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 30, 2024

Description

Changes

Possible Issues

Security Hotspots

petemill commented Oct 8, 2024 •

edited

Loading

fallaciousreasoning left a comment •

edited

Loading