Magentic-One Log Viewer + preview API

[husseinmozannar]

In this PR I make the following changes:

• multimodal web surfer saves screenshots with timestamps
• updated test of web surfer to handle deletion of screenshots
• update example.py script for magentic one to have hil mode flag and logs dir
• update loghandler to keep a logs list object (may be reduntant)
• created a MagenticOneHelper api
• create a logviewer for MagenticOneHelper logs
• changes to MagenticOne readme, @victordibia and I can complete it later
• removed TERMINATE and request halt ability of coder

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• [X ] I've made sure all auto checks have passed.

[husseinmozannar]

log_viewer is creating issues as it is a flask app, everything else should be okay

[gagb]

This is the log I got when I ran the example-- this looks buggy,

It also saved many html and png files in cwd -- is that intentional?

 .../autogen/python/packages/autogen-magentic-one (magentic_one_viz) $ python examples/example.py 
/workspaces/autogen/python/packages/autogen-magentic-one/examples/example.py:12: DeprecationWarning: DockerCommandLineCodeExecutor moved to autogen_ext.code_executors.DockerCommandLineCodeExecutor
  from autogen_ext.code_executor.docker_executor import DockerCommandLineCodeExecutor
/workspaces/autogen/python/.venv/lib/python3.12/site-packages/pydub/utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work
  warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)
[2024-11-01T23:46:46.417969], WebSurfer: Screenshot: screenshot_1730504805.png
User input ('exit' to quit):  hi

--------------------------------------------------------------------------- 
[2024-11-01T23:46:49.447624], UserProxy:

hi

--------------------------------------------------------------------------- 
[2024-11-01T23:46:52.715617], Orchestrator (thought):

Initial plan:

We are working to address the following user request:

hi


To answer this request we have assembled the following team:

WebSurfer: A helpful assistant with access to a web browser. Ask them to perform web searches, open pages, and interact with content (e.g., clicking links, scrolling the viewport, etc., filling in form fields, etc.) It can also summarize the entire page, or answer questions based on the content of the page. It can also be asked to sleep and wait for pages to load, in cases where the pages seem to be taking a while to load.
Coder: A helpful and general-purpose AI assistant that has strong language skills, Python skills, and Linux command line skills.
Executor: A agent for executing code
file_surfer: An agent that can handle local files.



Here is an initial fact sheet to consider:

1. GIVEN OR VERIFIED FACTS
   - The request given is "hi," which does not contain any specific facts or figures.

2. FACTS TO LOOK UP
   - None, as the request does not state any facts that require external verification.

3. FACTS TO DERIVE
   - None, since the request is simply a greeting and does not necessitate any logical deduction or computation.

4. EDUCATED GUESSES
   - The request might be an initial greeting to start a conversation or interaction.


Here is the plan to follow as best as possible:

- Acknowledge the greeting in the request by responding with a polite and friendly message.
- Ask if there's anything specific the user would like assistance with.
- Offer help or provide information about the team and capabilities if the user needs guidance on what kind of requests can be fulfilled.


--------------------------------------------------------------------------- 
[2024-11-01T23:46:57.860572], Orchestrator (thought):

Updated Ledger:
{
  "is_request_satisfied": {
    "reason": "The request was just a 'hi' which was acknowledged but waiting for further instructions.",
    "answer": false
  },
  "is_in_loop": {
    "reason": "There is no loop since there hasn't been repetition of requests or responses.",
    "answer": false
  },
  "is_progress_being_made": {
    "reason": "The initial greeting has been acknowledged, but further clarification or instruction is needed from the user.",
    "answer": true
  },
  "next_speaker": {
    "reason": "WebSurfer, Coder, and file_surfer don't seem relevant as of now. Executor is not suitable either.",
    "answer": "WebSurfer"
  },
  "instruction_or_question": {
    "reason": "We should engage the user by asking if there's anything specific they need.",
    "answer": "Please let me know if there's anything specific you need help with or information about our capabilities you would like."
  }
}

--------------------------------------------------------------------------- 
[2024-11-01T23:46:57.860805], Orchestrator (-> WebSurfer):

Please let me know if there's anything specific you need help with or information about our capabilities you would like.

--------------------------------------------------------------------------- 
[2024-11-01T23:46:57.861051], Orchestrator (thought):

Next speaker WebSurfer
[2024-11-01T23:46:58.866696], WebSurfer: Screenshot: screenshot_som_1730504818.png

--------------------------------------------------------------------------- 
[2024-11-01T23:47:01.443052], WebSurfer:

Hello! How can I assist you today?

--------------------------------------------------------------------------- 
[2024-11-01T23:47:05.684503], Orchestrator (thought):

Updated Ledger:
{
  "is_request_satisfied": {
    "reason": "The user's request was simply a greeting, which has been acknowledged. There is no further specific request to fulfill.",
    "answer": true
  },
  "is_in_loop": {
    "reason": "There is no indication of being in a loop as this was a simple, initial interaction.",
    "answer": false
  },
  "is_progress_being_made": {
    "reason": "Progress is being made as the initial greeting has been acknowledged, and there's an open invitation for further requests.",
    "answer": true
  },
  "next_speaker": {
    "reason": "Since the initial request was just a greeting and has been acknowledged, there is no need for further action unless the user elaborates.",
    "answer": "WebSurfer"
  },
  "instruction_or_question": {
    "reason": "To check if the user has any specific requests or needs assistance with something else.",
    "answer": "Please let us know if there's anything specific you need assistance with or if there's a particular topic you're interested in."
  }
}

--------------------------------------------------------------------------- 
[2024-11-01T23:47:05.684869], Orchestrator (thought):

Request satisfied.

--------------------------------------------------------------------------- 
[2024-11-01T23:47:06.460186], Orchestrator (final answer):


Hello! How can I assist you today? If you have any specific questions or need help with something, feel free to let me know.

--------------------------------------------------------------------------- 
[2024-11-01T23:47:06.461485], Orchestrator (termination condition):

No agent selected.
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x7879c780b9c0>
Traceback (most recent call last):
  File "/usr/local/python/3.12.1/lib/python3.12/asyncio/base_subprocess.py", line 126, in __del__
    self.close()
  File "/usr/local/python/3.12.1/lib/python3.12/asyncio/base_subprocess.py", line 104, in close
    proto.pipe.close()
  File "/usr/local/python/3.12.1/lib/python3.12/asyncio/unix_events.py", line 767, in close
    self.write_eof()
  File "/usr/local/python/3.12.1/lib/python3.12/asyncio/unix_events.py", line 753, in write_eof
    self._loop.call_soon(self._call_connection_lost, None)
  File "/usr/local/python/3.12.1/lib/python3.12/asyncio/base_events.py", line 792, in call_soon
    self._check_closed()
  File "/usr/local/python/3.12.1/lib/python3.12/asyncio/base_events.py", line 539, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

[husseinmozannar]

the log seems as expected, except for the runtime error, I can't replicate it from my side, can you try it again?

for saving many screenshots, this is intentional. We want to have screenshot at every step of the web browser navigation for display of the logs or creation of videos. There is an argument for where to store the logs

[afourney]

Test -- works well.
Discussed in meeting.