Python script to take mongodb backup

"""
@Author: Aseem Jain
@profile: https://www.linkedin.com/in/premaseem/

"""
import os
import pymongo

# configure credentials / db name
db_user = os.environ["MONGO_ATLAS_USER"]
db_pass = os.environ["MONGO_ATLAS_PASSWORD"]
db_name = "sample_mflix"

connection_string = f"mongodb+srv://{db_user}:{db_pass}@sharedcluster.lv3wx.mongodb.net/{db_name}?retryWrites=true&w=majority"

client = pymongo.MongoClient(connection_string)
db = client[db_name]

# create database back directory with db_name
os.makedirs(db_name, exist_ok=True)

# list all tables in database
tables = db.list_collection_names()

# dump all tables in db
for table in tables:
    print("exporting data for table", table )
    data = list(db[table].find())
    # write data in json file
    with open(f"{db.name}/{table}.json","w") as writer:
        writer.write(str(data))

exit(0)

How to execute mongo commands through shell

Lets say you need to find out which is the primary node in the mongo cluster in your shell script,

mongo –host 10.126.71.162 –username=userid –password=secret –eval “db.runCommand(‘ismaster’).primary”

This would quickly open the mongo shell and execute the commands, and give the output as the last line

MongoDB shell version v4.0.5
connecting to: mongodb://10.126.71.162:27017/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“65f2e5ab-aa97-4306-8a20-afd70a4ceeb0”) }
MongoDB server version: 4.0.5
10.126.71.162:27017

Dynamodb: How to run dynamodb locally

DynamoDb is a cloud managed server provided by AWS – Amazon web service. Technically, you cannot install and manage it on your own for your production system. However for testing purpose, AWS provides the ways to run it locally and avoid hitting cloud service all the time to save some money.

The easiest way is to pull docker image and run it locally

Follow below steps to run dynamodb locally

# pull image from docker hub provided by AWS – https://hub.docker.com/r/amazon/dynamodb-local/

docker pull amazon/dynamodb-local

# run image

docker run -p 8000:8000 amazon/dynamodb-local

Hit this url on browser  to test dynamodb is working

http://localhost:8000/shell/

MongoDB : History tracking micro service

Mongo DB local -> oplog.$main collection keeps records of all changes which happen in the primary database, which are eventually read by secondaries to catch up data or syncup data in the mongo replica set.

A micro service can be written, which can pull up data / delta form oplog.$main based on the name space ( defined db and collection) and can save that data in destination Db or audit DB.

Later on audit DB can the queried to fetch the history or log data pertaining to entity.

Architecture: How mongodb replication works and secondary node get data from primary node

Here is the simple diagram which explains how the middle man oplog watches/monitors primary and pull any new operation or changes happened on primary and records them in oplog, which eventually get read by all secondaries to update them selves.

1. insert operation happened in primary
2. Operation got logged in oplog ( which is part of local db – oplog.$main )
3. Secondary reads data from oplog and update them selves.

Transaction ACID properties

A database transaction is a sequence of actions that are treated as a single unit of work. These actions should either complete entirely or take no effect at all. Transaction management is an important part of RDBMS-oriented enterprise application to ensure data integrity and consistency. The concept of transactions can be described with the following four key properties described as ACID −

• Atomicity − A transaction should be treated as a single unit of operation, which means either the entire sequence of operations is successful or unsuccessful.
• Consistency − This represents the consistency of the referential integrity of the database, unique primary keys in tables, etc.
• Isolation − There may be many transaction processing with the same data set at the same time. Each transaction should be isolated from others to prevent data corruption.
• Durability − Once a transaction has completed, the results of this transaction have to be made permanent and cannot be erased from the database due to system failure.

MongoDB : How to write Stored Procedures in MongoDB

The first thing you need to know is that MongoDB stored procedures are written in JavaScript. This feels pretty strange at first, but in actual fact it’s far more satisfying to program in a real language than in any of those vendor-specific SQL-based concoctions. It’s also very powerful: MongoDB uses the same JavaScript runtime as FireFox, so you have pretty much every feature of the language, including closures, Iterators and XML support via E4X.

So in MongoDB, a stored procedure is really just a JavaScript function, which must be stored in a special collection named db.system.js. Imagine a simple function like so:

function addNumbers( x , y ) {
    return x + y;
}

To convert that to a stored proc, just insert it into db.system.js:

> db.system.js.save({_id:"addNumbers", value:function(x, y){ return x + y; }});

Stored procs can be viewed, altered and deleted in the same way as any document in any collection, as that’s all they are. So we can check that our code was successfully stored by using a simple find():

> db.system.js.find()
{ "_id" : "addNumbers", "value" : function cf__3__f_(x, y) {
    return x + y;
} }

Ignore the machine-generated function name. That looks good, and we can start to actually use the proc. That’s done using db.eval():

> db.eval('addNumbers(17, 25)');
42

That’s all pretty simple, I think.

Python : Dictionary comprehension

Before updating a no sql or mongo doc you need to ensure that you are not adding keys which are not part of initial doc then dictionary comprehension is really helpful.

What it means, is you are able to create a dict (json object) dynamically in single line of code using loops, condition, construction every thing together.

Single line comprehension

items_to_update = {key: value for key, value in payload.items() if key in req_items}

 

multi line interpretation

if key in req_items:

….for key, value in payload.items():

…….. dictionary[key]=value

Detailed example of created update doc dic before inserting in mongo db

try:
    req_items = ["property1","property2"]
    items_to_update = {key: value for key, value in payload.items() if key in req_items}
  
    if items_to_update:
        logger.info("Updating device collection with {}".format(items_to_update))
        self.db.update({"filter": filter}, {"$set": items_to_update}, multi=True)
        logger.info("Successfully updated product details")
except PyMongoError as e:
    logger.critical('Pymongo Exception {}'.format(str(e)))
except Exception as e:
    logger.critical('Exception {}'.format(str(e)))

 

asd

MongoDb : Add user to DB

use myDb
db.createUser(
   {
     user: "accountUser",
     pwd: "password",
     roles: [ "readWrite", "dbAdmin" ]
   }
)

Login with admin role, use your db and create user with role.

Mongodb – can’t login via -u root -p bitnami

Are you trying to login like this?

mongo admin --username root --password YOUR_PASSWORD

where YOUR_PASSWORD is the one you can see in your AWS system log:

Setting Bitnami application password to 'YOUR_PASSWORD'

If this doesn’t work, you can also try resetting MongoDB password:

http://wiki.bitnami.com/Components/mongoDB?highlight=mongo#How_to_reset_the_MongoDB_root_password.3f