Hello ,
Can anyone please help me to add the RoundRobin logic for offset limit ? Only using 10 threads i should be able to create offset value and assign offset value in roundrobin way between 10threads . Am fetching 3million records from couchbase which internally causing n1ql timeout . This will solve my issue <could anyone please help me on this ?
query := "select count(1) as count from `" + configuration.BucketName + "` where APPID='" + configuration.APPID + "'"
var countVariable int = 0
countVariable = countVariable + 1
fmt.Println("countVariable value: ", countVariable)
rows, err := cluster.Query(query, &gocb.QueryOptions{})
if err != nil {
panic(err)
}
var row map[string]interface{} //To fetch n1ql query output
for rows.Next() {
err := rows.Row(&row)
if err != nil {
panic(err)
}
}
countTotal := int64(row["count"].(float64))
fmt.Println("countTotal num value : ", countTotal)
if countTotal == 0 {
fmt.Println("No rows returned")
os.Exit(3)
}
float_offset_count := math.Ceil(float64(int(countTotal) / configuration.Limit))
offset_count := int(float_offset_count) + 1
fmt.Println("offset_count :", offset_count)
for i := 0; i < 10; i++ { // should be only 10 threads
wg.Add(1)
//fmt.Println("Goroutine :",i)
offset_limit := configuration.Limit * i // limit is 20000
go n1qlFetch(i, &wg, offset_limit) // This function contains n1ql query which fetches 3 to 4 ,million records
}
Am trying implement the logic as below
total documents - 2000000
limit :20000
offset : 2000000/20000 = 100
threads :10
so offset value will be as below :
0
20000
40000
60000
80000
100000
.
.
.
.
2000000
thread 1 [0,20000,40000,60000,80000,100000,120000,140000,160000,180000,200000] --> 0 to 2lac records
thread 2 [220000,240000,260000,280000,300000,320000,340000,360000,380000,400000,420000]
thread 3 ""should be divided same as above
thread 4 ""should be divided same as above
thread 5 ""should be divided same as above
thread 6 ""should be divided same as above
thread 7 ""should be divided same as above
thread 8 ""should be divided same as above
thread 9 ""should be divided same as above
thread 10 ""should be divided same as above
for each ele in offsetlist:
offset = ele