2.0 Pixel and layer separation training
There are two main steps in the layer separation training process: separating the layers manually in Pixel, and producing models based on that training.
For this step you will need:
- A folio image.
When you run it for the first time, the Pixel workflow is built like so:
Important
If you've already run Pixel and Paco trainer once, your workflow will look different.
It's also possible to run this workflow without the Background Removal job. This is covered here.
The resources are assigned like so:
You can now run the job. It may take some time, so don't worry. You can check the jobs' status under the 'Run Jobs' tab of the workflow run page. We do recommend refreshing the page every now and then, as Rodan won't always immediately reflect completed jobs/workflow runs. When it's complete, you'll see the row look like this:
Once the job row says "Ready for input," as above, clicking the row will add a new section to the column on the right hand side of your screen, at the bottom of which will be several buttons. Click on "OPEN" to open up Pixel.js and begin classifying.
This will open a new window on your computer--pixel.js!
Caution
TROUBLESHOOTING: pixel.js is big. It's highly recommend using a desktop if possible, or otherwise being very patient.
A popup window should have now appeared. In it, you should see your folio image with its background completely black (see image below for an example). This is normal! The 'Background Removal' job has done its job and removed the background; all of that black surface is effectively empty space.
If the popup window hasn't appeared, ensure you have allowed popups from rodan2.simssa and try again. In the new popup you should see the image you uploaded at the top of the screen you should see the title of your project or file.
Caution
TROUBLESHOOTING: If this title is still "CDN Salzinnes", don't be alarmed, this is an ongoing feature/bug we're aware of. If this is still up in May, please go to out Github page and comment here that this is still present.
At the top of your screen you will see three boxes; click on the black square to enter editing mode (refer to picture below). A tutorial explaining how to navigate and use pixel will pop up. To skip the tutorial, click on 'Got it!' at the bottom of the pop-up.
Once you've closed the tutorial pop-up, if you've assigned your ports exactly as in the instructions above, all of the foreground elements (symbols, staff lines, text) should have turned blue; this is because Pixel has assigned everything to the foreground to layer 1. You can now access the toolbox by scrolling in the thin gap between the image viewer and the edge of the browser. It may be a thin white or grey line. Alternatively, enable 'view scrollbars' in your browser and manually scroll down to the bottom, where you should see the toolbox (if you are in editing mode).
Double click into each relevant layer to rename it. To ensure that this documentation makes the most possible sense, it's recommended that you assign resources like so: Layer 1 is symbols (including notes and other musical elements like clefs, accidentals, and custodes), Layer 2 is staff lines, and Layer 3 is text.
It's very important when training the computer to provide it with all the examples it needs to identify any elements that you want to include in each layer. For example, if your manuscript has both black staves and red staves, you must include both colours of staff in your training for layer 2. If you only include black staves, the computer won't know to include red staves in the layer as well. The same goes for different colours of text and neumes, or different types of ink.
The first thing you should do is select the area you will work on. YOU DO NOT NEED TO WORK THE ENTIRE MANUSCRIPT IMAGE. NOT ONLY WILL THIS TAKE YOU FOREVER IT WILL TAKE FOREVER TO PROCESS.
Having highlighted the relevant area, you should now see a vague colour overlay of the area. This is your boundary.
Caution
TROUBLESHOOTING: Highlighting/brushing areas will have periods of exceptional lag if you are using a computer with insufficient memory. If you experience lag, move in incremental spaces until you become more familiar. It is doable, but slow.
Because everything is now in Layer 1, you have to transfer the text and the staff lines to their own layer (but you don't have to do anything with the glyphs, because they're already in their own layer!). To do this, follow these steps:
- Click on Layer 1 and then click on the 'Select' tool;
- Click and drag to select a region of the folio with elements that should belong in either the staff layer or the text layer;
- Press
command+Xorcontrol+X(elements in your selected region will no longer be blue); - Click on the layer you want the selected elements to be in (layer 2 for staff, layer 3 for text);
- Press
command+Vorcontrol+V; - The elements in the selected layer should reappear in the correct colour.
You can also manually "paint" over the elements that should belong to each layer by clicking on the 'Brush' or 'Rectangle' tools. Remember that this information will then be used by the computer for future layer separation, so the more consistent and precise you are, the better the computer will perform. To highlight elements in a specific layer, click on the layer you want to edit.
Important
It's very possible that the 'Background Removal' job will have left elements of the book, like the edges of the cover or other pages, in the foreground. If that's the case, it's important to include some of this in your selected region and delete it from Layer 1. If not, the computer will be taught that those elements are symbols, and this will be annoying later.
If having every element belong to Layer 1 when you open Pixel is confusing, you can avoid this by assigning the 'Empty Layer' output port of 'Background Removal' to all Layer inputs of 'Pixel' and then painting everything manually. You can also remove the Background Removal job entirely; you can read about that here.
Tip
SLOW IS SMOOTH, SMOOTH IS FAST. This process will take some time. The slower you move and more attention to detail you can devote, the better your sample will be for training.
Once the correction is done, it can be submitted back to Rodan for further processing by clicking on "Submit to Rodan" at the bottom of the page. The resulting ZIP file will appear under the 'Resources' tab.
Caution
TROUBLESHOOTING: At this time (2024-02-22), nothing will happen when clicking this button. DO NOT CLOSE PIXEL. It will close by itself after a moment. If it doesn't close, it's possible to change the window to view the Rodan Project page, navigate to the Resources tab and refresh the page.
Instead of submitting directly to Rodan, it's possible to download the layers as individual and a ZIP files. This is recommended to cover all bases and be certain. The files can then be uploaded as resources to the Project.
Congratulations, you've annotated and sorted your first image!
You should now have several items in the resources window, among them a zip file. Look, don't open—it's huge. Next step is training.
For this step, you will need:
- The ZIP file (or files) produced by your Pixel workflow runs.
This workflow contains only one job: Training model for Patchwise Analysis of Music Document, Training (note: this is distinct from Training model for Patchwise Analysis of Music Document, Classifying, so make sure double check you have the right one.) This job is also known as Paco Trainer.
Create a new workflow and upload the Training model for Patchwise Analysis of Music Document, Training job. When it first appears, the job will look like this:
Input and output ports should be added like so (ports can be added by right-clicking on the job and selecting "Ports"):
Make sure as you're assigning your ports that the number of model outputs matches the number of layers. The standard practice is to use three layers to produce three models: a symbol model, a staff model, and a text model. It's possible however to make as many layers and models as you want, as long as their numbers and contents match exactly. If you're producing more than three models, you'll have to add the corresponding output ports in the Paco Trainer job.
If you have multiple pages which you trained, separated, and marked, you can upload multiple ZIP files (produced by the previous Pixel.js download stage) to the Training model for Patchwise Analysis of Music Document, Training input port. You do NOT need to add more model outputs; you only need as many models as you have layers.
When run, this will take a long time to process. The more zipped files you assigned the longer this will take (hours). Plan accordingly!
Eventually, you will see the following:
Caution
TROUBLESHOOTING: If your first runs fail, navigate to the Run Jobs tab, click the row that failed so that the 'RUN JOB DETAILS' window pops up, and scroll to the very bottom of the 'Error Details' box. This should detail exactly want went wrong, and you'll be able to go back, fix it, and run it again. See below:
Caution
MORE TROUBLESHOOTING: If the job finishes quickly, something is most probably wrong. When something has gone wrong with the job, its status will report 'Failed'. Follow steps mentioned above to locate reason for failure, then navigate back to workflows and run the job again (likely you have too few or too many models).
Important
The number of folios needed to train reliable models varies from manuscript to manuscript, depending on the density of the glyphs on each folio. A manuscript with few staves per page and relatively big neumes will need 8-10 pages, whereas a manuscript with 15+ staves per page and small neumes might need only 3-4.
Once you've run the Paco Trainer once, you can use the models produced to get the computer to help you. Using the Fast Pixelwise Analysis of Music Document, Classifying the computer will use those models to make an initial layer separation on your folio image; instead of highlighting everything, you now only have to make corrections to what the computer has done. The new workflow looks like this:
And the resources are assigned like this:
Once the workflow has run and Pixel is ready for input, once in editing mode you should see the folio image with three colours already on it: blue for Layer 1 (symbols), red for layer 2 (staves), and yellow for layer 3 (text). All you have to do now is make corrections to this initial layer assignement.
Repeat steps 2.3 and 2.4 until your models allow the computer to make a satisfactory layer separation on a new folio image!
Without the Background Removal job, the workflow is very simple:
You don't need to create any input ports for the layers; once you open Pixel.js and click on the black square, you'll be prompted to indicate how many layers you will need, excluding the background layer (a reminder that this tutorial follows the three-layer system: layer 1 for glyphs, layer 2 for staff lines, and layer 3 for text):
If you want the individual layers to be output, as well as the ZIP file, it is important to add those layer output ports to the job.