Binary package CRLF problem [RE-OPENED]

Hey there, I think got a big problem here with a tcp server routine, I use a custom scanner to find CRLF, I tried both “\r\n” and as byte 0x0D, 0x0A, it works, the problem is that sometimes I get packages with 0D0A in the middle of the package. Below a packet with 0D0A in the middle;

78 78 1f 12 11 02 0d 0a 04 1e c9 01 c5 3d 4a 05 4d b6 8c 00 38 14 02 d4 06 0c 5a 00 7a 79 00 12 ff 0e 0d 0a 

And below my scanner code;

// dropCR drops a terminal \r from the data.
func dropCR(data []byte) []byte {
	if len(data) > 0 && data[len(data)-1] == '\r' {
		return data[0 : len(data)-1]
	}
	return data
}

func ScanCRLF(data []byte, atEOF bool) (advance int, token []byte, err error) {
	if atEOF && len(data) == 0 {
		return 0, nil, nil
	}
	if i := bytes.Index(data, []byte{'\r', '\n'}); i >= 0 {
		// We have a full newline-terminated line.
		return i + 2, dropCR(data[0:i]), nil
	}
	// If we're at EOF, we have a final, non-terminated line. Return it.
	if atEOF {
		return len(data), dropCR(data), nil
	}
	// Request more data.
	return 0, nil, nil
}

Below the usage;

func (c *Client) listen() {
	reader := bufio.NewReader(c.conn)
	scanner := bufio.NewScanner(reader)
	scanner.Split(ScanCRLF)
	for scanner.Scan() {
		c.Server.onNewMessage(c, strings.ToUpper(hex.EncodeToString(scanner.Bytes())+"0d0a"))
	}

}

Any one knows a way to fix it?

Thanks all…

So, to clarify your question, you only want to recognise the 0d0a at the end of the package (packet?)?

How do you define the end of the packet?

Yes, I need to recognize that on the end of the packet.

Tnx.

A TCP stream (and an io.Reader) are a stream of data, how do you determine the end of one packet?

1 Like

It’s not me who determines the end of the data, that packet comes from a GPS tracker, and the end of the data according to their documentation, is 0d0a, beggining is 7878 and the end is 0d0a, I even did a crc checksum in order to verify the data integrity and it’s ok.

Then what’s the problem? If 0d0a appears in the stream, that is the end of message. Are you saying 0d0a appears midway through a packet?

Friend, I guess you didn’t understand my question, maybe because of my english.

This is not a valid packet;

78 78 1f 12 11 02 0d 0a

This is a valid one;

78 78 1f 12 11 02 0d 0a 04 1e c9 01 c5 3d 4a 05 4d b6 8c 00 38 14 02 d4 06 0c 5a 00 7a 79 00 12 ff 0e 0d 0a

Explaining the packet;

7878 = beginning
1f 12 11 02 0d 0a 04 1e c9 01 c5 3d 4a 05 4d b6 8c 00 38 14 02 d4 06 0c 5a 00 7a 79 00 12 = data
ff 0e = crc16 checksum
0d0a = end

If you put 1f1211020d0a041ec901c53d4a054db68c00381402d4060c5a007a790012 in any crc16 calc tool, you’ll see it’s a valid packet, so that packages could have 0d0a inside the data, because data is not converted into string as the end of the packet is.

If you put it into this tool, select HEX and click “Calc CRC-16”, you will see that the checksum matches the CRC-16/X-25 result;
http://crccalc.com/

[EDIT]

PS. I have all of this working for 6 years in a Delphi service application, and it works. If there is any way to get if there’s nothing after “0d0a”… That would be a way to know this is the packet end.

Tnx.

Wow, what a complex data format. So the logic is

  1. if you find 0d 0a, then the two bytes preceeding that are the crc16 of the entire string from 78 78 til 0d 0a.
  2. If the checksum doesn’t match then either this is not the end of the line, keep reading or the data so far is corrupt. In which case ???
1 Like

What does your delphi code look like, it might simpler to start from there.

This packet brings all the information about the tracker, every single byte inside this part of the packet, should not be converted to string, I mean, some bytes in this part are converted into binary string, decimal, etc… So, the 0d0a at the end of the packet means “\r\n”, in the middle of the packet it’s different. Explaining the “1f1211020d0a041ec901c53d4a054db68c00381402d4060c5a007a790012” data;

1f = Packet length converted to decimal.
12 = Message type.
11020d0a041e = GPS date and time

I’m not gonna explain the whole packet now, otherwise my post would be too big, but below you have an explanation about “gps date and time” data, where 0d0a appears;

Every single byte should be converted to decimal;

11 = Year
02 =Month
0d = Day
0a = Hour
04 = Minute
1e = Second

Where

11 = 17
02 = 02
0d = 13
0a = 10
04 = 04
1e = 30

17-02-13 10:04:30

Tnx.

procedure TServicoAccurateGT06.IdTCPServer1Execute(AContext: TIdContext);
  var
    ConAux: PConexao;
    RxBuf: TIdBytes;
    crcCalculado, crcPacote, InfoSerialNum: String;
    ThreadID: Cardinal;

    Rec, ack, Imei, strCmd, TipoMsg, GPSValido, Longitude, Latitude,
    Velocidade, Ignicao, AntiFurto, Saida1, Saida2, Saida3, Saida4,
    Panico, Entrada1, Entrada2, Entrada3, Entrada4,
    Direcao, ConexaoGPRS, FalhaAntenaGPS,
    VelocidadeMaximaExcedida, BateriaCarregando, FalhaBateriaInterna, FalhaAlimentacaoPrincipal,
    Movendo, Tensao, Hodometro, Horimetro, IndiceUltimaPosicao, TerminalInfo, NivelSinalGSM, CursoStatus: String;
    DataHora: TDatetime;
    IdMotorista, IdVeiculo, IndicePosicao, GrauAngulo: Integer;
    UltimaPosicao, PosicaoAtual: TPonto;
    Query, queryUltimaPosicao: TADOQuery;
    RegistraALerta, ackTerminalBloqueio, ackTerminalDesbloqueio, LatitudeNegativa, LongitudeNegativa: Boolean;
begin
  try
      RxBuf := nil;
      with AContext.Connection.IOHandler do
      begin
        CheckForDataOnSource(10);

        if not InputBufferIsEmpty then
        begin
          ThreadID := TIdYarnOfThread(AContext.Yarn).Thread.ThreadID;
          InputBuffer.ExtractToBytes(RxBuf);
          Rec := ToHex(RxBuf);

We use a library called Indy, I need only that code to get received packets…

Right, this is good information. As this format contains a length header, you probably don’t want to treat it like a string of bytes with 0d 0a as a terminator instead do something like this (not tested)

var header [3]byte
err := ioutil.ReadFull(header[:])
check(err)

// check that header[0] and header[1] contain 78 
length := header[2] // i'm not sure how to do this conversion
body := make([]byte, length)
err := ioutil.ReadFull(body)
check(err)

body, trailer := body[:len(body)-2], body[len(body)-2:] // trim of the trailer
// check that trailer[0] == 0d and trailer[1] == 0a
4 Likes

I’m new to GO. Could you explain what everyline does on that solution you posted? I’m confusing about this line;

body, trailer := body[:len(body)-2], body[len(body)-2:]

Splits the body to body (2 bytes before the end) and trailer (last 2 bytes)

This one worked perfectly, the only problem is when it finds a 0a in the middle of the string;

// Read client data from channel
func (c *Client) listen() {
	reader := bufio.NewReader(c.conn)
	scanner := bufio.NewScanner(reader)
	var rst []byte
	for scanner.Scan() {
		bytes := scanner.Bytes()

		if bytes[0] == 0x78 && bytes[1] == 0x78 {
			tampac := int(bytes[2]) + 1
			rst = make([]byte, tampac)
			for i := 2; i < tampac+2; i++ {
				rst[i-2] = bytes[i]
			}
		}
		c.Server.onNewMessage(c, strings.ToUpper(hex.EncodeToString(rst)))
	}
}

The length + 1 I’m using is because the crc calc needs the packet length byte too, I gotta include it, so,if I my packet is

78781f1211020d0a041ec901c53d4a054db68c00381402d4060c5a007a790012ff0e0d0a

I need

1f1211020d0a041ec901c53d4a054db68c00381402d4060c5a007a790012ff0e

Maybe searching for ‘\0’?

Stop using bufio.Scanner, this will not work for your application

1 Like

Then post an example on here… I can’t find example or any information in nowhere… Could you help me?

Tnx.

Binary package CRLF problem [RE-OPENED]

Hey @leogazio,

Here’s something I threw together for you, but I haven’t read all of the other posts so if I’m forgetting something then let me know. Otherwise it should give you a good starting point.

package main

import (
	"bufio"
	"bytes"
	"fmt"
	"io"
	"log"
)

func main() {
	packet := []byte{0x78, 0x78, 0x1f, 0x12, 0x11, 0x02, 0x0d, 0x0a, 0x04, 0x1e, 0xc9, 0x01, 0xc5, 0x3d, 0x4a, 0x05, 0x4d, 0xb6, 0x8c, 0x00, 0x38, 0x14, 0x02, 0xd4, 0x06, 0x0c, 0x5a, 0x00, 0x7a, 0x79, 0x00, 0x12, 0xff, 0x0e, 0x0d, 0x0a}

	var hdr [3]byte
	r := bufio.NewReader(bytes.NewReader(packet))

	// Read into the header.
	n, err := r.Read(hdr[:])

	// Make sure 3 bytes were read into the header.
	if n < 3 || err != nil && err != io.EOF {
		log.Fatalln("invalid packet header")
	}

	// Make sure first 2 bytes are 0x78.
	if hdr[0] != 0x78 || hdr[1] != 0x78 {
		log.Fatalln("packet header start is not 0x78 0x78")
	}

	// Get packet length.
	ln := int(hdr[2])

	data := make([]byte, ln)
	n, err = r.Read(data)
	if n != ln || err != nil {
		log.Fatalln("error reading packet data, %v", err)
	}

	// Extract the crc from the last 2 bytes of the data.
	crc := data[len(data)-2:]

	// Read the final 2 bytes from the packet's end.
	end := make([]byte, 2)
	n, err = r.Read(end)
	if n != 2 || err != nil {
		log.Fatalln("error reading packet data, %v", err)
	}

	// Make sure the 2 end bytes are correct.
	if end[0] != 0x0d || end[1] != 0x0a {
		log.Fatalln("packet end is not 0x0d 0x0a")
	}

	fmt.Printf("header: %x\ndata: %x\ncrc: %x\nend: %x\n", hdr, data, crc, end)
}

Result:

header: 78781f data: 1211020d0a041ec901c53d4a054db68c00381402d4060c5a007a790012ff0e crc: ff0e end: 0d0a

2 Likes

Hey Benjamin, guys, It’s almost done, that code on playground works, but when I put in into my TCP, the first condition doing header validation returns true and stops execution, but at the end I get the right package, below, how is my code now;

// Read client data from channel
func (c *Client) listen() {
	reader := bufio.NewReader(c.conn)

	for {
		var hdr [3]byte
		//r := bufio.NewReader(bytes.NewReader(reader))
		r := bufio.NewReader(reader)

		// Read into the header.
		n, err := r.Read(hdr[:])

		// Make sure 3 bytes were read into the header.
		if n < 3 || err != nil && err != io.EOF {
			log.Fatalln("invalid packet header")
		}

		// Make sure first 2 bytes are byte 78.
		if hdr[0] != 0x78 && hdr[1] != 0x78 {
			log.Fatalln("packet header start is not 78 78")
		}

		// Get packet length.
		ln := int(hdr[2])

		data := make([]byte, ln)
		n, err = r.Read(data)
		if n != ln || err != nil {
			log.Fatalln("error reading packet data, %v", err)
		}

		// Extract the crc from the last 2 bytes of the data.
		//	crc := data[len(data)-2:]

		// Read the final 2 bytes from the packet's end.
		end := make([]byte, 2)
		n, err = r.Read(end)
		if n != n || err != nil && err != io.EOF {
			log.Fatalln("error reading packet data, %v", err)
		}

		// Make sure the 2 end bytes are correct.
		if end[0] != 0x0d && end[1] != 0x0a {
			log.Fatalln("packet end is not 0x0d 0x0a")
		}

		strpacote := fmt.Sprintf("%x%x%x", hdr, data, end)
		c.Server.onNewMessage(c, strings.ToUpper(strpacote))
	}
}

I even tried to wireshark it, below package used, but it happens even with the same tested package on here. What should it be?

78780a1346004300010014cb230d0a

It’s almost working, tnx.