IDE doesn't properly handle byte order mark

[McNeight]

http://en.wikipedia.org/wiki/Byte_order_mark
The UTF-8 representation of the BOM is the byte sequence 0xEF, 0xBB, 0xBF. A text editor or web browser interpreting the text as ISO-8859-1 or CP1252 will display the characters  for this.

Adding the byte order mark to a UTF-8 encoded library.properties file causes that library to be ignored by the IDE. Removing the three bytes at the beginning of the file "restores" the library.

Adding the byte order mark to a UTF-8 encoded header file causes the following compilation error:

In file included from ReceiveJ1939Messages.ino:28:
C:\Users\mcneightne\Documents\Arduino\libraries\CAN_Library\src/CAN.h:1: error: stray '\357' in program
C:\Users\mcneightne\Documents\Arduino\libraries\CAN_Library\src/CAN.h:1: error: stray '\273' in program
C:\Users\mcneightne\Documents\Arduino\libraries\CAN_Library\src/CAN.h:1: error: stray '\277' in program

(with \357, \273, \277 being the octal representations of 0xEF, 0xBB, 0xBF)

I'm not sure if the IDE can do anything about passing the characters on to the compiler, but it should at least be able to handle the characters in the library.properties file.

-Neil

[ffissore]

@McNeight is this the output of gcc or the IDE? Can you paste the whole output?

[ffissore]

If it's gcc's, then I guess you hit this [1] and you should report this upstream

[1] http://stackoverflow.com/questions/7899795/is-it-possible-to-get-gcc-to-compile-utf-8-with-bom-source-files

[McNeight]

@ffissore I missed the notification that you had picked up this issue, however the issue still exists in 1.6.4.

There were two issues, which I should have split out, but the one that can be fixed here is with the IDE. The steps to recreate the bug are simple:

1. Take a library with a valid library.properties file and see that it is listed under File -> Examples
2. Close the IDE and edit the library.properties file to include the BOM at the beginning of the file
3. Restart the IDE and see that the library is no longer listed under File -> Examples

[Chris--A]

I guess the IDE team simply need to add a note to the documentation that the library.properties/internal configurations are ANSI only.

[McNeight]

@Chris--A I believe that the IDE handles Unicode characters in the library.properties file correctly. I haven't looked at the actual code, but I suspect that the problem lies in the parsing of the first key=value pair. If it expects one of 9 well defined strings at the beginning of the file, and instead finds the three characters used for the BOM, then I suspect it would mark the file and library as invalid and move on.

[ffissore]

Indeed the whole IDE internally works with UTF8 and reads UTF8 files with no problems. But BOM is a 3 byte garbage we don't expect. Don't use BOM. On Linux you can use bomstrip to remove it