selected group jumps position when deselecting

[Icyretsz]

I am building a Canva-like app using React and Fabric.js. Users can click on objects, which opens a sidebar menu where they can modify the fill color of the selected object(s). The logic I want to implement is as follows:

When the user clicks on an object and opens the color menu, a sidebar appears, allowing them to modify the fill color.
If the user deselects the object by clicking on the canvas once, only the color menu should close, and the object(s) should remain selected.
A second click on the canvas should actually deselect the object(s).
This works fine for a single object. However, when there are multiple objects selected (using Fabric.js's ActiveSelection), and the user clicks on the canvas to deselect them, the objects "jump" to a different position after the second click.
Below is my current implementation:

`
canvas.on('before:selection:cleared', () => {

            if (selectedObjects.length > 0) {

                flushSync(() => {

                    localSelectedObjectsRef.current = canvas.getActiveObject();

                });

            }

        })

        canvas.on('selection:cleared', () => {

            if (activeTool[1] !== "") {

                setActiveTool(activeTool[0], "")

                if (activeTool[0] === "Select") {

                    setExpanded(false)

                }

                if (localSelectedObjectsRef.current !== null) {

                    canvas.setActiveObject(localSelectedObjectsRef.current);

                    canvas.renderAll();

                }
            } else if (activeTool[1] === "") {

                setSelectedObjects([])

            }

        })`

Explaination:
The first event listener is for storing current active objects.
The second event listener is for handling the logic behind the menus appearance: activeTool is an array of 2 elements, you shouldn't worry about the activeTool[0], it is activeTool[1] that controls the appearance of the menus. When there is a menu being opened (activeTool[1] !== ""), make it empty and then if the active objects storage is not null, make them active with setActiveObject. Those logic is for the first click.
On the second click (which activeTool[1] is now empty), reset the activeObject array

Current Behavior:
On the first click outside the object, the sidebar menu closes as expected, and the object remains selected.
On the second click outside the object:
For a single object, the behavior is as expected: the object is deselected.
For multiple selected objects (a group), the group of objects jumps to a different position after deselection.

A video for better demonstration of the problem:

Screen.Recording.2024-09-09.at.19.37.15.mov