Can't figure out how to parse buffered array

Getting Help
1

This is how the data comes in

[2 68 52 50 58 49 48 32]
[32 48 32 32 48 32 32 48]
[48 49 48 49 48 32 32 32]
[32 3 120]
[2 68 52 50 58 49 49 32]
[32 48 32 32 48 32 32 48]
[48 49 48 49 48 32 32 32]
[32 3 121]
[2 68 52 50 58 49 50 32]
[32 48 32 32 48 32 32 48]
[48 49 48 49 48 32 32 32]
[32 3 122]
[2 68 52 50 58 49 51 32]
[32 48 32 32 48 32 32 48]
[48 49 48 49 48 32 32 32]
[32 3 123]
[2 68 52 50 58 49 52 32]
[32 48 32 32 48 32 32 48]
[48 49 48 48 48 32 32 32]
[32 3 125]
[2 68 52 50 58 49 52 32]
[32 48 32 32 48 32 32 48]
[48 49 51 48 48 32 32 32]
[32 3 126 2 68 52 50 58 49 52 32 32 48 32 32 48]
[32 32 48 48 49 51 48 48]
[32 32 32 32 3 126]
[2 68 52 50 58 49 52 32]
[32 48 32 32 48 32 32 48]
[48 49 51 48 48 32 32 32]
[32 3 126]
[2 68 52 50 58 49 52 32]
[32 48 32 32 48 32 32 48]
[48 49 48 49 48 32 32 32]
[32 3 124 2 68 52 50 58 49 52 32 32 48 32 32 48]
[32 32 48 48 49 48 49 48]
[32 32 32 32 3 124]
[2 68 52 50 58 49 53 32]
[32 48 32 32 48 32 32 48]
[48 49 48 49 48 32 32 32]
[32 3 125]

This is how I demonstrated this

conn, err := net.Dial("tcp", "192.168.2.64:9001")

	if err != nil {
		fmt.Println("Connection not established")
	}

	fmt.Println(conn.LocalAddr())

	for {
		buffer := make([]byte, 1024)

		bytesRead, err := conn.Read(buffer)

		if err != nil {
			fmt.Println(err)
		}

		fmt.Println(buffer[:bytesRead])

	}

This is the program I’m using to try and parse it.

package main

import (
	"bufio"
	"fmt"
	"io"
	"net"
)

func calculateXORChecksum(data []byte) byte {
	var checksum byte
	for _, b := range data {
		checksum ^= b
	}
	return checksum
}

func findByte(data []byte) int {

	for index, aByte := range data {
		if aByte == 2 {
			return index
		}
	}
	return -1
}
func readSTXETXWithChecksum(conn net.Conn) ([]byte, byte, error) {
	reader := bufio.NewReader(conn)

	// Read until we get the ETX (ASCII 3) character
	message, err := reader.ReadBytes(3)
	if err != nil {
		return []byte(""), 0, err
	}

	checksumByte, err := reader.ReadByte()
	if err != nil {
		return []byte(""), 0, err

	}

	data := append(message, checksumByte) // All characters including the checksum

	return data, checksumByte, nil
}

func main() {
	conn, err := net.Dial("tcp", "192.168.2.64:9001")

	if err != nil {
		fmt.Println("Connection not established")
	}

	fmt.Println(conn.LocalAddr())

	for {
		fmt.Println()

		data, _, err := readSTXETXWithChecksum(conn)
		if err == io.EOF {
			fmt.Println("Connection closed by the peer")
		} else if err != nil {
			fmt.Println("Error reading message:", err)
		} else {
			fmt.Println("Received data:", []byte(data))

			fmt.Println("Received data:", string([]byte(data)[:len(data)-1]))

			fmt.Println()
		}

	}

}

The problem is when I get bytes array such as [32 3 126 2 68 52 50 58 49 52 32 32 48 32 32 48] anything after 3 and checksum byte gets discarded. This is due to the fact that in each itteration of the for loop a new scanner is declared. However if I declare a reader outside the for loop the buffer array is gonna get filled up at some point and there may be missing data as a new one is declared when it is full.

I feel quite stupid for not being able to solve something that seems simple.
I tried something manual but cant get it right and feel like it is not the right approach

buffer := make([]byte, 1024)
	var aLine []byte

	var remaining []byte
	for {

		numberRead, err := conn.Read(buffer)

		if err != nil {
			fmt.Println("the error")
		}

		if len(remaining) != 0 {
			aLine = append(remaining, buffer[:numberRead]...)
		}
		// fmt.Println(aLine)

		foundEnd := findByte(buffer[:numberRead])
		if foundEnd != -1 {

			aLine = append(aLine, buffer[:foundEnd+1]...)
			fmt.Println(aLine)
			aLine = nil
			if numberRead > foundEnd+1 {
				leftover := buffer[foundEnd+1 : numberRead]
				remaining = append(remaining, leftover...)
			}

		} else {
			aLine = append(aLine, buffer[:numberRead]...)
		}
2

Apparently it is correct to define Reader externally.
It’s not clear to me what you’re worried about? Don’t know how you are going to lose data? The ReadBytes function you are using does not cause this problem.
If you still have questions, please write down the reason, I can’t know what you’re worried about.

3

I might be getting confused due to an earlier problem I encountered.

	f, err := os.Open("./serial-formated.txt")
	if err != nil {
		fmt.Println("err")
	}

	defer f.Close()
	reader := bufio.NewReaderSize(f, 4082)

	startByte := 2
	endByte := 3
	for {

		n := 26

		// Read n bytes from standard input
		buf := make([]byte, n)
		numRead, err := reader.Read(buf)
		readChar := []byte(buf[:numRead])

I had to read every 26 bytes and so had to make sure the reader array size is divisible by 26 (4082 here). Otherwise sometimes it wouldn’t read 26 bytes everytime.

so when I am reading data from the tcp connection I’m worried what will happen if in
reader := bufio.NewReaderSize(conn, 1024)
The buffer array is filled up to 1022 bytes and then it receives 6 more bytes.

4

If you want to make sure you want to read enough bytes, why not use io. ReadFull？
io.ReadFull(reader, buf)

Interface io. Reader will only determine the upper limit based on the size of the buf you pass in, bufio. Reader is internally buffered, and when you call Read, it will pre-read a portion of its own buffer, and then deliver it from the buffer to your buf. About this, it is possible to read the source code, which is not difficult.

// Read reads data into p.
// It returns the number of bytes read into p.
// The bytes are taken from at most one Read on the underlying [Reader],
// hence n may be less than len(p).
// To read exactly len(p) bytes, use io.ReadFull(b, p).
// If the underlying [Reader] can return a non-zero count with io.EOF,
// then this Read method can do so as well; see the [io.Reader] docs.
func (b *Reader) Read(p []byte) (n int, err error) {
	n = len(p)
	if n == 0 {
		if b.Buffered() > 0 {
			return 0, nil
		}
		return 0, b.readErr()
	}
	if b.r == b.w {
		if b.err != nil {
			return 0, b.readErr()
		}
		if len(p) >= len(b.buf) {
			// Large read, empty buffer.
			// Read directly into p to avoid copy.
			n, b.err = b.rd.Read(p)
			if n < 0 {
				panic(errNegativeRead)
			}
			if n > 0 {
				b.lastByte = int(p[n-1])
				b.lastRuneSize = -1
			}
			return n, b.readErr()
		}
		// One read.
		// Do not use b.fill, which will loop.
		b.r = 0
		b.w = 0
		n, b.err = b.rd.Read(b.buf)
		if n < 0 {
			panic(errNegativeRead)
		}
		if n == 0 {
			return 0, b.readErr()
		}
		b.w += n
	}

	// copy as much as we can
	// Note: if the slice panics here, it is probably because
	// the underlying reader returned a bad count. See issue 49795.
	n = copy(p, b.buf[b.r:b.w])
	b.r += n
	b.lastByte = int(b.buf[b.r-1])
	b.lastRuneSize = -1
	return n, nil
}

	n = copy(p, b.buf[b.r:b.w])
	b.r += n
	b.lastByte = int(b.buf[b.r-1])
	b.lastRuneSize = -1
	return n, nil

Visible, in bufio with buffer size. The reader consumes the contents of the buffer first. In your case, when a non-26 multiple is used, a non-26 length will be returned, because bufio. The reader’s buffer is not a multiple of 26.
The obvious conclusion is that there will be no data loss.

1 Like
5

Thanks yeah that will solve it for my read every 26 bytes problem.

so when I am reading data from the tcp connection I’m worried what will happen if in
reader := bufio.NewReaderSize(conn, 1024)
The buffer array is filled up to 1022 bytes and then it receives 6 more bytes.

Do you have any insights into this part?

6

Nothing happens, data doesn’t disappear into thin air.

buf:=make([]byte,1022)

//r := bufio.Reader // 1024 size and full
n, _ = r.Read(buf)
// n will 1022
n, _ = r.Read(buf)
// n will 1024 - 1022 = 2

n, _ = r.Read(buf)
// next read src ; n <= 1022

Note, I have you to remind you a little bit (maybe you know this), for tcp conn, it’s streaming, io.Reader interface does not guarantee that you will be able to read the buffer-sized data every time.

7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.