MongoDB-How to perform Insert operation?

Getting Help
1

Hi Team,

I have an array which stores huge data and need to insert those data into MongoDB.

I am able to achieve this using below code.But taking more time.Is ther anyother way to push huge array data into MongoDB?

session, err := mgo.Dial(“localhost”)
if err != nil {
panic(err)
}
defer session.Close()
// Optional. Switch the session to a monotonic behavior.
session.SetMode(mgo.Monotonic, true)

c := session.DB("Test").C("Indicators")

for i := 0; i < len(HeadDet); i++ {
    err = c.Insert(HeadDet[i])
}
if err != nil {
    log.Fatal(err)
}

I have refered below link
https://labix.org/mgo

Thanks,
Alex

3

The documentation for mgo says you can insert one or more documents with insert . link.

There can probably be a problem as the inserted array can end up with one single id (as explained here).

Mongo DB intself has a db.collection.insertmany() method. But not sure mgo implements it.

Sorry I did not help much.

1 Like
4

I’m not sure if mgo uses similar terminology as other DB drivers I know from other languages, but I’d take a look at the Bulk methods.

Also, it seems as if you should search for another driver, as mgo seems unmaintained:

mgo package - gopkg.in/mgo.v2 - Go Packages

#########################################################

THIS DRIVER IS UNMAINTAINED! See here for details:

mgo/README.md at v2-unstable · go-mgo/mgo · GitHub

#########################################################

5

I switched to this fork in one project

because official driver is still in alpha stage.

Unfortunately, this driver doesn‘t support context methods for cancellation, while official does.

7

Thanks for input.I am able store around 30 lakhs records in within 3 min.

using below
//HeadDet having 30 lakhs records

docs := make([]interface{}, len(HeadDet))
for i := range HeadDet{
docs[i] = HeadDet[i]
}
//inserting docs into data base
collection.Insert(docs…)

But I want insert array data in the fraction of seconds instead of 3 min. Iterating array data is taking time. Is there any other way to reduce time to fraction of seconds?.

Thanks.
Alex

8

Use workers (goroutines) and separate mgo session in each (multiple connections). Look at mgo session. Copy(). Then it depends on your Mongo server how fast it is going to be.

1 Like
9

Thanks.Is there any example for this?

10

I am trying to implement goroutine to insert data into mongo db as suggested. But it is working for 50 to 100 records. But huge records it didn’t work. Can anyone help on this?

11

What error message are you getting? Do you have any more information? “It does not work” is not something that helps us help you. What is not working? How is it not working?

12

I am not getting any error.It is inserting into MongoDB with 50 to 100 records.If i am trying with huge records it is not inserting into DB.I am using below code.

package test

    import (
     "os"

    "gopkg.in/mgo.v2"
      )

   var Db *mgo.Session

  type DbLink struct {
  DbName string
 }

func NewDbSession(url string) {
var e error   

Db, e = mgo.Dial(url)

if e != nil {
	panic(e)
	os.Exit(-1)
}
}

func PerformInsertOperation() {

NewDbSession("localhost")
session := Db.Copy()
//defer session.Close()

col := session.DB("TEST").C("HeadDet")
ch := make(chan bool)

go func(doc []interface{}) {

	col.Insert(doc...)

	ch <- true

}(HeadDet)

 }

Thanks,
Alex

13

The docs show that col.Insert() returns an error, which you are dismissing. Just do something like this to know what’s wrong:

err := col.Insert(doc...)
if err != nil {
   fmt.Errorf("insert error: %s\n", err) //
}

The lesson is: never dismiss errors, always handle them.

1 Like
14

Thanks. I have handled the error as well. But it is not inserting into db.No Error was throwing as well. Same code I have checked with some dummy data which would generate 50 to 100 records and same records have been inserted in DB without any issues. Only large data(3 Million) having problem.

15

How do you tell it’s not working then? How do you know there’s an error? Does your program indeed end and then you check your DB to see there was no data inserted? Do logs show any attempt on inserting data? etc.

The example code you presented is not complete and you are not giving a lot of information for us to help. Please give us more details and also start debugging things, throw a couple of prints here and there, maybe even use a debugger (e.g., https://github.com/derekparker/delve).

On another note, though that particular call may have not thrown an error, I’ll insist that you should always check and handle errors.

2 Likes
16

How many goroutines you run at the same time? Does everyone has it’s own session copy? Do you close sessions?

17

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.