Loading Data Into MongoDB With Go: A Practical Guide

In this guide, we'll explore how to load data into MongoDB using Go, focusing on a practical implementation that includes adding a custom field during the loading process. We will leverage a dataset, implement the loading process in a Go file, and ensure that each document includes an additional 'password' field with a value derived from the user ID.

Prerequisites

Before we dive in, make sure you have the following set up:

• Go: Ensure Go is installed and configured on your system. You can download it from the official Go website.

• MongoDB: You need a MongoDB instance running, either locally or on a remote server. You can download MongoDB from the official MongoDB website.

• MongoDB Go Driver: Install the MongoDB Go driver using the following command:

  go get go.mongodb.org/mongo-driver/mongo
  

Choosing a Dataset

For this guide, let's use a sample dataset of user information in JSON format. This dataset will include fields like userID, name, email, and age. Here’s an example of what the data might look like:

[
  {
    "userID": 1,
    "name": "John Doe",
    "email": "john.doe@example.com",
    "age": 30
  },
  {
    "userID": 2,
    "name": "Jane Smith",
    "email": "jane.smith@example.com",
    "age": 25
  },
  {
    "userID": 3,
    "name": "Alice Johnson",
    "email": "alice.johnson@example.com",
    "age": 28
  }
]

Save this data in a file named users.json. This file will serve as our data source for loading into MongoDB.

Implementing the Data Loading Process in Go

Now, let's create a Go program to load this data into MongoDB. The program will read the users.json file, parse the JSON data, and insert each user document into a MongoDB collection. Additionally, it will add a password field to each document before inserting it.

Setting Up the Go File

Create a new file named load_data.go and add the following code:

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"log"
	"os"

	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
)

// User struct to represent the structure of the user data
type User struct {
	UserID int    `json:"userID"`
	Name   string `json:"name"`
	Email  string `json:"email"`
	Age    int    `json:"age"`
	Password string `json:"password,omitempty"` // Add password field
}

func main() {
	// MongoDB connection string
	mongoURI := os.Getenv("MONGO_URI")
	// Database and collection names
	const dbName = "mydatabase"
	const collectionName = "users"

	// Load MongoDB URI from environment variables
	// mongoURI := os.Getenv("MONGO_URI")
	// if mongoURI == "" {
	// 	log.Fatal("MONGO_URI environment variable not set")
	// }
	// Set client options
	clientOptions := options.Client().ApplyURI(mongoURI)

	// Connect to MongoDB
	client, err := mongo.Connect(context.TODO(), clientOptions)
	if err != nil {
		log.Fatal(err)
	}

	// Check the connection
	err = client.Ping(context.TODO(), nil)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("Connected to MongoDB!")

	// Get a handle to the database and collection
	collection := client.Database(dbName).Collection(collectionName)

	// Read data from the JSON file
	data, err := ioutil.ReadFile("users.json")
	if err != nil {
		log.Fatal(err)
	}

	// Unmarshal the JSON data into a slice of User structs
	var users []User
	err = json.Unmarshal(data, &users)
	if err != nil {
		log.Fatal(err)
	}

	// Prepare the data for insertion
	var documents []interface{}
	for _, user := range users {
		// Add the password field
		user.Password = fmt.Sprintf("password%d", user.UserID)
		documents = append(documents, user)
	}

	// Insert the data into the MongoDB collection
	insertManyResult, err := collection.InsertMany(context.TODO(), documents)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("Inserted multiple documents: ", insertManyResult.InsertedIDs)

	// Disconnect from MongoDB
	err = client.Disconnect(context.TODO())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("Connection to MongoDB closed.")
}

Explanation of the Code

1. Import Packages: We import necessary packages, including context, encoding/json, fmt, io/ioutil, log, os, and the MongoDB Go driver.
2. User Struct: The User struct defines the structure of our user data. Note the Password field, which will store the custom password we generate.
3. MongoDB Connection:
   • We read the MongoDB URI from the environment variables using os.Getenv("MONGO_URI").
   • We create a MongoDB client using mongo.Connect.
   • We ping the MongoDB server to ensure the connection is successful.
4. Read Data from JSON File:
   • We read the users.json file using ioutil.ReadFile.
   • We unmarshal the JSON data into a slice of User structs.
5. Prepare Data for Insertion:
   • We iterate through the slice of User structs.
   • For each user, we create a custom password by concatenating the string password with the userID.
   • We append each user document to a slice of interfaces to prepare it for insertion.
6. Insert Data into MongoDB:
   • We use the InsertMany method to insert the data into the MongoDB collection.
   • We print the IDs of the inserted documents.
7. Disconnect from MongoDB:
   • We disconnect from the MongoDB server.

Setting the MongoDB URI

Make sure to set the MONGO_URI environment variable with your MongoDB connection string. For example:

export MONGO_URI="mongodb://localhost:27017"

Replace `