Unexpected bugs due to data encoding or others things like breakpoint in csv

[kikislater]

I write a csv from code using a programming language like python. Then I send the csv to another user. I don't know in which editor the user modifies the csv but the modified file results in an R breakpoint (red dot, screenshot available below) only visible in RStudio (gedit, geany, vscodium, nano do not display these breakpoints).

File is available here: nosysakatia_zenodo-rawdata.csv @juldebar : this is your csv ^^

Finally, when I run a geoflow workflow with this csv, an error is generated. Removing the breakpoints allows the workflow to work, of course. To avoid this kind of error, it would be good to have a control to remove these breakpoints.

Finally, this made me think of something else. I'm used to receiving csvs or dbfs from shapefiles with different types of user and I then run into problems in different programming languages. It is common to include a dict to convert some data to other data. Perhaps by implementing this removal of "R breakpoints", it would be good to have a dictionary to handle bad encodings!

Example of hardcoding dict:

                    if "%20" in dkan_tag_name:
                        dkan_tag_name = dkan_tag_name.replace("%20", '-')
                    # Parenthèses
                    if "%28" in dkan_tag_name:
                        dkan_tag_name = dkan_tag_name.replace("%28", '')
                    if "%29" in dkan_tag_name:
                        dkan_tag_name = dkan_tag_name.replace("%29", '')
                    # Virgules
                    if "%2C" in dkan_tag_name:
                        dkan_tag_name = dkan_tag_name.replace("%2C", '')
                    # Apostrophes
                    if "%28" in dkan_tag_name:
                        dkan_tag_name = dkan_tag_name.replace("%28", '')
                    # non standard dash
                    if "%E2%80%93" in dkan_tag_name:
                        dkan_tag_name = dkan_tag_name.replace("%E2%80%93", '-')
                    # Apostrophe
                    if "&#039" in dkan_tag_name:
                        dkan_tag_name = dkan_tag_name.replace("'", '')

Sylvain

[eblondel]

I've no idea where the user modified the file, and I think this where you should first investigate the matter.

We use the standard readr package read_csv method to read CSV as inputs to geoflows. I tried to isolate your file and read it with the csv entity handler available in geoflow at inst/metadata/entity/entity_handler_csv.R using a recent version of readr, and I had no issue to read entities. I suggest you look at your readr version, it might be an old one?

This example let you attempt reading the entity from your csv:

require(geoflow)

config = geoflow::add_config_utils(list()) #empty config (only with logger utils)

source = "path to your csv file"

handlers = geoflow::list_entity_handlers(raw = T)
csv_entity_handler = handlers[sapply(handlers, function(h){h$id == "csv"})][[1]]

handle_entities_df <- source(system.file("metadata/entity", "entity_handler_df.R", package = "geoflow"))$value
entities <- handle_entities_df(csv_entity_handler, source, config)

In case you spot out errors downstream in the workflow, let me know

[kikislater]

readr 2.1.4 or 2.1.5 and I can see the breakpoints in the csv ... It seems to be more of an RStudio issue, so maybe not related to readr. BTW i didn't tried to launch geoflow with these versions. The question is why RStudio shows the breakpoints in the csv ...

After this hard-to-find problem, I switched to R in client mode and didn't encounter the same problem.

We have had other files from other people for example with the same behaviour ...

[eblondel]

Thanks, this is weird indeed. So it works in native R, but not in Rstudio. I found this https://community.rstudio.com/t/non-breaking-spaces-highlighted-even-when-show-whitespace-characters-is-not-selected/51477 it might be related.

What is not clear to me is what are the errors you have geoflow? Is it really a running code problem? may it be only a "display" issue in Rstudio?