CSV, characters problem with write


(Miha) #1

Hello

i am reading some data from REST and then I am writing them to CSV file.
Data has special characters which are seend in csv file correctly but when CSV file is opened, the characters are not presented correctly.

Characters like “č,ć,š,ž”
My code:
writer := csv.NewWriter(file)
defer writer.Flush()
writer.UseCRLF = true
err := writer.Write(value)

thank you!
miha


(Curtis Allyn Green) #2

I’m not sure about this but it sounds like a UTF-8 encoding problem on your end. Might help you figure out where to look.


(Johan Dahl) #3

Hi

The problem is Excel. It defaults to open csv files with wrong encoding. There is 3 solutions

  1. Import from inside Excel https://support.office.com/en-us/article/text-import-wizard-c5b02af6-fda1-4440-899f-f78bafe41857?ui=en-US&rs=en-US&ad=US
  2. Change extension to .txt and right click it and select open with Excel and then will Excel show the same import guide
  3. Convert the file to the format Excel will import automatically. It is a UTF16LE file with the first line is a 16-bit word 0XFFFE followed by “sep=\t\n” and then the rest of the file as UTF16LE (little endian). Maybe you could switch the two first bytes and use UTF16BE (big endian)

(Eric Lindblad) #4

What OS are you on?

What program is used to open the “CSV” file?


(Miha) #5

The program that is reading this CSV file is on windows server.
OS where golang runs is linux (docker -> golang 1.8)


(Norbert Melzer) #6

And which program is it?

Does it require a certain encoding or can you change the encoding to match with what go emits?

You should probably update. 1.11 is current and 1.8 is probably not supported by many libraries anymore, usually only current and last minors are supported by the libraries, that means: 1.11 and 1.10.


(Miha) #7

hello @NobbZ for sure it require certain ecoding, UTF8. But from documentation part of library, it should be UTF8. I tried to find how to change encoding but with no luck…

I will try to upgrade golang


(Norbert Melzer) #8

Well, we do not know where your data comes from, maybe it’s already in the wrong format? Depending on how you read and write the data might get converted but does not necessarily…

So you need to show more code perhaps even a simplified input.


(Eric Lindblad) #9

I’ll add a link here to the Kermit project.

http://kermitproject.org/utf8.html


(Eric Lindblad) #10

this ticket is more pursuant to UNIX or DOS end-of-line style, but the Git for Windows implementations of the Unix utilities ‘file’ and ‘od’ are demonstrated therein


(Eric Lindblad) #11

It would be nice to see K95 enabled for OpenSSH, possibly also for Heimdal instead of Kerberos, and for OpenSSL.

Frank da Cruz gave this comment in an e-mail last year.

“So if Kermit 95 has any future, it is as an SSH client. Of course it is still useful with serial ports and modems, but that is a very small niche in the XXI Century.”

Any volunteers?


(Miha) #12

Hello

i am using librarie: https://github.com/Genert/pipedrive-api/

result, _, err := client.Persons.List(context.Background()) and then save with above code

func CsvGen(data [][]string) {

file, err := os.Create("result.csv")
checkError("Cannot create file", err)
defer file.Close()

writer := csv.NewWriter(file)
defer writer.Flush()
writer.UseCRLF = true

for _, value := range data {
	//	fmt.Println("before for:", len(value))
	for {
		if len(value) >= 5 {
			break
		} else {
			value = append(value, "")
			//fmt.Println(value)
		}
		//fmt.Println("izven loopa:", len(value))

	}

	err := writer.Write(value)
	checkError("Cannot write to file", err)

}

}


(Norbert Melzer) #13

Sorry, I do not know that API or library.

Please show the the fmt.Printf("%#v", []byte(item)) representation of an broken example, how you’d expect to see it and what you see instead.

I’m still the opinion that the encoding in your datasource is not UTF-8.


(Eric Lindblad) #14

after running ‘go.exe build’ on this code saved as a UTF-8 encoded file, then running the resultant executable and redirecting its output to a .csv file, when opening that .csv file in notepad.exe, the last names are rendered correctly

package main
import "fmt"
func main() {
	fmt.Println("Lončar,Tkalčić")
}

the resultant .csv file incidentally also being UTF-8 encoded


(Eric Lindblad) #15

You might open an issue in the pipedrive-api repository.