MongoDB-How to perform Insert operation?


(Alex Roz) #1

Hi Team,

I have an array which stores huge data and need to insert those data into MongoDB.

I am able to achieve this using below code.But taking more time.Is ther anyother way to push huge array data into MongoDB?

session, err := mgo.Dial(“localhost”)
if err != nil {
panic(err)
}
defer session.Close()
// Optional. Switch the session to a monotonic behavior.
session.SetMode(mgo.Monotonic, true)

c := session.DB("Test").C("Indicators")

for i := 0; i < len(HeadDet); i++ {
    err = c.Insert(HeadDet[i])
}
if err != nil {
    log.Fatal(err)
}

I have refered below link
https://labix.org/mgo

Thanks,
Alex


(Michael De Barros) #3

The documentation for mgo says you can insert one or more documents with insert . link.

There can probably be a problem as the inserted array can end up with one single id (as explained here).

Mongo DB intself has a db.collection.insertmany() method. But not sure mgo implements it.

Sorry I did not help much.


(Norbert Melzer) #4

I’m not sure if mgo uses similar terminology as other DB drivers I know from other languages, but I’d take a look at the Bulk methods.

Also, it seems as if you should search for another driver, as mgo seems unmaintained:

https://godoc.org/gopkg.in/mgo.v2

#########################################################

THIS DRIVER IS UNMAINTAINED! See here for details:

https://github.com/go-mgo/mgo/blob/v2-unstable/README.md

#########################################################


(Boban Acimovic) #5

I switched to this fork in one project

because official driver is still in alpha stage.

Unfortunately, this driver doesn‘t support context methods for cancellation, while official does.


(Alex Roz) #7

Thanks for input.I am able store around 30 lakhs records in within 3 min.

using below
//HeadDet having 30 lakhs records

docs := make([]interface{}, len(HeadDet))
for i := range HeadDet{
docs[i] = HeadDet[i]
}
//inserting docs into data base
collection.Insert(docs…)

But I want insert array data in the fraction of seconds instead of 3 min. Iterating array data is taking time. Is there any other way to reduce time to fraction of seconds?.

Thanks.
Alex


(Boban Acimovic) #8

Use workers (goroutines) and separate mgo session in each (multiple connections). Look at mgo session. Copy(). Then it depends on your Mongo server how fast it is going to be.


(Alex Roz) #9

Thanks.Is there any example for this?


(Alex Roz) #10

I am trying to implement goroutine to insert data into mongo db as suggested. But it is working for 50 to 100 records. But huge records it didn’t work. Can anyone help on this?


(Kristoffer Berdal) #11

What error message are you getting? Do you have any more information? “It does not work” is not something that helps us help you. What is not working? How is it not working?


(Alex Roz) #12

I am not getting any error.It is inserting into MongoDB with 50 to 100 records.If i am trying with huge records it is not inserting into DB.I am using below code.

package test

    import (
     "os"

    "gopkg.in/mgo.v2"
      )

   var Db *mgo.Session

  type DbLink struct {
  DbName string
 }

func NewDbSession(url string) {
var e error   

Db, e = mgo.Dial(url)

if e != nil {
	panic(e)
	os.Exit(-1)
}
}

func PerformInsertOperation() {

NewDbSession("localhost")
session := Db.Copy()
//defer session.Close()

col := session.DB("TEST").C("HeadDet")
ch := make(chan bool)

go func(doc []interface{}) {

	col.Insert(doc...)

	ch <- true

}(HeadDet)

 }

Thanks,
Alex


(Ignacio Gómez) #13

The docs show that col.Insert() returns an error, which you are dismissing. Just do something like this to know what’s wrong:

err := col.Insert(doc...)
if err != nil {
   fmt.Errorf("insert error: %s\n", err) //
}

The lesson is: never dismiss errors, always handle them.


(Alex Roz) #14

Thanks. I have handled the error as well. But it is not inserting into db.No Error was throwing as well. Same code I have checked with some dummy data which would generate 50 to 100 records and same records have been inserted in DB without any issues. Only large data(3 Million) having problem.


(Ignacio Gómez) #15

How do you tell it’s not working then? How do you know there’s an error? Does your program indeed end and then you check your DB to see there was no data inserted? Do logs show any attempt on inserting data? etc.

The example code you presented is not complete and you are not giving a lot of information for us to help. Please give us more details and also start debugging things, throw a couple of prints here and there, maybe even use a debugger (e.g., https://github.com/derekparker/delve).

On another note, though that particular call may have not thrown an error, I’ll insist that you should always check and handle errors.


(Boban Acimovic) #16

How many goroutines you run at the same time? Does everyone has it’s own session copy? Do you close sessions?