MongoDB (NoSQL) and Python

[ Why | What | When | How & 3 ways to connect]

MongoDB (NoSQL database) + Python (Back-end OOP language)

If you are new to backend development or beginner - articulated it in detail with sample data set & code snippets, by illustrating one of the well-required & real-time scenarios is CRUD operations which is implemented in most of the web or desktop (GUI) applications using the MongoDB with Python.

Three available ways to communicate with MongoDB database to store, retrieve/fetch/read, update and delete data.

Let’s discuss briefly about database and their varieties. It may feel that we already know about databases! but it’s very essential to analyze the task to-do!

So, let’s dive into its WHY, WHAT, WHEN and HOW

WHY we need database/s:

Databases, play a vital role in our real-life & real-world applications as we are in digital era and ever-increasing requirements of data and its increasing availability!

So, we need tools to work with data and fortunately we have wide-range of tools available to store and interact with data.

Tools are incorporated depending on the type of application/software to be built and of course affordability! SQL and NoSQL - both provide us with their merits and limits. That’s why the selection of tools is crucial which again depends on the application under construction!

Types of database languages:

i) SQL - Structured Query Language

ii) NoSQL - Non SQL

• Differences between NoSQL and SQL types:

Some of the main differences between NoSQL and SQL.

This is an overview of SQL with NoSQL. If you are interested in learning more about SQL with Python, you can read Interact with MySQL Database article, in which I have explained it in more detail. MySQL - relational database and SQL - language to interact with this database.

Next, let’s dive into NoSQL - MongoDB

WHAT is NoSQL and MongoDB:

• NoSQL: It’s Non Structured Query Language, helps to define dynamic schema for databases that provides flexibility for defining keys/columns and are horizontally scalable.
• MongoDB: It’s non-relational database, helps to store data in key-value pairs. It’s general purpose and document-based database in short.

WHEN for MongoDB:

• Real-time applications - Ex: eBay, MetLife, Shutterfly, Adhar etc
• Gaming applications - Ex: EA, FIFA online game
• Content Management Systems, Full-stack web applications, Products data management systems
• Mobile apps
• Real-time Analytics
• Its implementation in IoT applications
• It can be used in combo with SQL database

To read more about its use-cases - https://www.mongodb.com/use-cases

Examples of SQL and NoSQL databases:

Terminologies/building blocks in MongoDB: Database, Collection, Document and Keys/Fields.

Terminologies comparision between NoSQL and SQL

Structure of data storage in both types:

Data in Table format in SQL (left image — MySQL Workbench) and NoSQL (right image — MongoDB Compass).

Data Types in MongoDB: Below are the common data types supported in MongoDB which are available in Object Oriented Programming languages too. However one of the specific data type to MongoDB is ObjectID that let’s to store Document’s unique ID.

Common data types in MongoDB.

Tools and Language: MongoDB Community server, MongoDB Compass, Jupyter Notebook, VS Code and Python3 for this CRUD task.

HOW - connect to MongoDB database

3 ways to communicate with MongoDB:

i) Mongo Shell

ii) Python

iii) MongoDB Compass

Now, let’s communicate to MongoDB using the method (ii) - Python3. It’s one of the back-end programming language to interact with database. it’s a super friendly thus high-level OOP and interpreted language. It’s versatile & helps to build & deploy wide-variety of applications. Especially in Artificial Intelligence & Data Science world, it plays a vital role!

Methods (i) and (iii) are explained in high-level. You can refer to the Compass screenshot further below to help visualize the database, collection & documents that gets created when we execute the scripts shown next.

You can execute the Python scripts via one of these two options: In this article, described with both of these options.

Option-1. Jupyter Notebook

Option-2. PyCharm (IDE) or VS Code or Atom (Text editors)

With Python script, we again have two options to communicate with MongoDB, they are Pymongo (driver) and MongoEngine (Document-Object Mapper) - similar to Django ORM.

Below scripts are shown with Pymongo driver i.e., Python module - provides the functions & methods to interact with MongoDB database.

Once the Python3 is installed, then install Pymongo as below:

$ pip3 install pymongo  # for MAC OS
> pip install pymongo   # for Windows OS

Notes:

• MongoDB stores its documents in BSON format. BSON is Binary JSON (JavaScript Object Notation). So, the documents are binary format of JSON objects.
• MongoDB - provides the flexibility of designing database with dynamic schema, so there’s no JOIN here. However, Serialization (to map objects to BSON document that can be saved in MongoDB) and Deserialization (other way round - representing an object from BSON document) are available.
• MongoDB server available in both Community and Enterprise versions. To begin with & learning purpose, Community version will suffice. However, paid version is available for deployment.
• Official installation links for these tools are provided in the Resources section, further below.

Prerequisites to communicate with MongoDB: Once you have installed the above tools and language, ensure below servers are running on your machine.

i) MongoDB community server

ii) Jupyter Notebook server

Next, enter the command (sudo mongod) in terminal and hit enter, then it will expect you to open the Compass or Shell for interaction. I often open Compass tool.

To start the MongoDB server:

Terminal shell to start MongoDB server.

For PC (Windows OS), type the below command in CMD and hit enter.

> mongod

To start the Jupyter Notebook server:

Firstly, install the Jupyter Notebook by writing the below command and hit enter in Terminal (MAC OS) or CMD (Windows OS) respectively:

$ pip3 install notebook  # for MAC OS
> pip install notebook   # for Windows OS

Note: Jupyter Notebook can also be installed via tool called Anaconda i.e., stand-alone product helps to install other tools such as Jupyter Lab, Notebook, VS Code, Python, PyCharm, RStudio etc under one roof! However, when you install them individually, you would learn more!

Then, write the below command in either Terminal or CMD and hit enter. The current working directory gets opened in the default browser of your machine.

$ jupyter notebook

To open the Jupyter Notebook in the browser of your choice, you can add additional tags as below and hit enter:

$ jupyter notebook --browser any

Output will be as below, next copy the highlighted localhost line & paste it in the browser’s tab of your choice:

To start the Jupyter Notebook server.

To stop the Notebook server, hit ctrl+c in the keyboard and type ‘y’ in the terminal to stop it.

Once you have started the above two servers, now you are ready for writing CRUD statements. Let’s see them in action, one-by-one!

The sample data set:

This is the custom sample data set incorporated.

Note: We can store List/Array or Dictionary as a value to the key in MongoDB.

Option-1. Python with Jupyter Notebook

Now, create a file and enter the name of your choice, ex: mongodb_crud_python using the UI of the Jupyter Notebook. This file gets saved with .ipynb extension in the current working directory (folder). Then install & import required module - pymongo as below and write the statements shown to interact with MongoDB server. However, atleast one document needs to be inserted to view the database & collection in the Compass.

.ipynb is file extension for Jupyter Notebook.

Module/library can be installed via Jupyter Notebook using PIP package manager.

Please note here, data saved as dictionaries and list of dictionaries in variables - doc_1, docs, test_docs and test_doc:

Data saved in variables.

If you don’t pass in the id key, MongoDB will automatically generate one for you. You can notice this in the below screenshot from Compass, that gets stored in ObjectID data type and highlighted in red.

Then to CREATE Documents in the database:

To insert one or more documents.

READ or Fetch Documents: Script to fetch one document and displayed its respective output.

To find one document.

To fetch specified number of documents with limit().

To fetch all documents in the collection (aka Table).

When find() or find({}) methods are used, it’s advised to store output in variable since they return cursor object which requires iterating one-by-one to display results.

UPDATE Documents: With UPDATE methods, we can add-in new key if it doesn’t already exists in the document.

To update one or more documents.

DELETE Documents:

To delete one or more documents.

Note: Jupyter Notebook is available in two flavours: Jupyter Lab and Notebook. Here what you see are from classic Notebook. It’s a web browser based editor & stands for Julia, Python and R languages.

You can visualize the above created or modified MongoDB Documents in Notebook via Compass tool.

Documents in Compass and Array/List passed as value to the keys ‘shift’ and ‘trains_for’.

It also helps us to view that, in MongoDB databases - not necessarily all the keys to be presented in all the documents. You can notice, ‘coach’ key is not there in first document but it’s there in second document, so that document specific keys can be added, which makes it dynamic!

Overview of method (i) - Mongo Shell:

After its installation, in terminal - type in the below command and hit enter, then it gets connected to MongoDB server’s local instance.

$ mongo

And its output will be as below:

Screenshot of Mongo Shell

CRUD statements above written in Python and Notebook can be executed in Mongo Shell and its syntax examples are below:

$ db.collection.insertOne() # To insert one document as the name says.
$ db.collection.insertMany()
$ db.collection.updateOne()
$ db.collection.updateMany() and so on . . .

To export data via Mongo Shell, you can use command-line tool called mongoexport and pass in the options to export documents to JSON or CSV formats. To read further - https://docs.mongodb.com/database-tools/mongoexport/

If you are interested in writing & executing these scripts using Mongo Shell - https://docs.mongodb.com/manual/mongo/

Overview of method (iii) - MongoDB Compass (GUI tool):

All the Python statements explained in the Notebook above can also be performed using Compass tool i.e., to create database, create collection and add-in documents/data, read, update or delete them and so on. You can view its screenshot from above for how it looks.

Compass too helps to export data in JSON or CSV formats. After exporting to JSON, you can view it in the text editor, which is an array of JSON objects:

JSON export from Compass.

An alternative tool to Compass is MongoDB Atlas, it’s cloud-based tool accessible via browser that requires hosted URL and is paid version however you get to explore Atlas if you have an account @ MongoDB!

Option-2: Python and VS Code

You can also use other text editors of your choice such as Atom or Sublime.

Screenshots of Python scripts written in VS Code for this CRUD task:

Part-1: Database is ‘mongodb_workshop’ and collection is ‘coach_details’ gets created.

Part-2: Description for each statement via comments are highlighted in green.

Part-3: find(), delete_one(), delete_many() & update_one() methods.

Part-4: To create another collection for the testing purpose, can be deleted with drop() method.

Resources:

i) To install MongoDB tools (MongoDB Server, Compass, Shell, Atlas) - visit https://www.mongodb.com/ and https://www.mongodb.com/try/download/community

ii) Python3 - https://www.python.org/downloads/

iii) VS Code - https://code.visualstudio.com/

iv) Anaconda — https://www.anaconda.com/products/individual

Summary:

• MongoDB can be used in combination with relational databases, means both NoSQL and SQL can be used in an application.
• Its syntax for inserting key-value pairs are same as JSON objects - keys to be enclosed with double quotes.

Hope you enjoyed this article and thank you for reading . . .