Write CSV with encoding ISO8859-1 return error


(Trond Kandal) #1

Hello

The problem is that I am supposed to support writing a CSV response with text encoding both as UTF-8 and ISO8859-1
This does not seem to work as planned.

When I try to stream:

… encoding: rune not supported by encoding.

To set up the csv-writer I use the following code:

// w is http.ResponseWriter
var conv io.Writer = w
if encoding == altEncoding {
    conv = charmap.ISO8859_1.NewEncoder().Writer(w)
}
csvWriter := csv.NewWriter(conv)
csvWriter.Comma = separator

As long as I use UTF-8, everything seems ok.

Any help is appreciated!


(Jakob Borg) #2

What are you trying to write? Not all characters in UTF-8 can be represented in ISO 8859-1.


(Trond Kandal) #3

Yes, that is awful truth about ISO8859-1 when converting from UTF-8 to ISO8859-1.
My data are names of persons and many of them are international, and as far as I am able to understand, those are the cause of this problem.


(Jakob Borg) #4

That’s indeed quite likely. ISO 8859-1 covers western european scripts. Outside of that you need a better character set.


(Trond Kandal) #5

Thank you very much, Sir!

Well, I guess, I have to go back to my client and say: “Nope, not possible!” :wink:


(Norbert Melzer) #6

If its mostly names, you can try to ask the original data source for transcriptions or to create it computationally. But transcribing or dropping ISO8859-1 are probably the only options.


(Eric Lindblad) #7

Can you keep your output files in UTF-8, and then on those use iconv?

sample text ‘naïveté’

iconv -f UTF-8 -t ISO8859-1 file1 > file2

uxterm cat file1, and xterm cat file2


(Trond Kandal) #8

Yes, I will probably drop the whole thing.

Than you, Sir


(Trond Kandal) #9

Yes, I have that option too

Thank you, Sir!


(Norbert Melzer) #10

By using inconv on the terminal rather than a library wrapping it, you are just defering the problem. Unicode knows more than a million “characters”, ISO encodings can encode just 256. You’ll always have a loss of information when not using UTF-8/16/32.


(Eric Lindblad) #11

see also iconv’s use of //TRANSLIT

maybe a filter could be written for the characters not found in ISO8859-1 which might eventually appear in your “names of persons”

a couple of forked GitHub repositories as Golang bindings to libiconv

https://github.com/go-iconv/iconv

https://github.com/mattn/go-iconv


(Eric Lindblad) #12

I am curious what the client intends the duplicate ISO8859-1 encoded files for.

Might those be for use in a certain editor or terminal?