MongoDB : History tracking micro service

Mongo DB local -> oplog.$main collection keeps records of all changes which happen in the primary database, which are eventually read by secondaries to catch up data or syncup data in the mongo replica set.

A micro service can be written, which can pull up data / delta form oplog.$main based on the name space ( defined db and collection) and can save that data in destination Db or audit DB.

Later on audit DB can the queried to fetch the history or log data pertaining to entity.

audit log sequence diagram


Architecture: How mongodb replication works and secondary node get data from primary node

Here is the simple diagram which explains how the middle man oplog watches/monitors primary and pull any new operation or changes happened on primary and records them in oplog, which eventually get read by all secondaries to update them selves.

  1. insert operation happened in primary
  2. Operation got logged in oplog ( which is part of local db – oplog.$main )
  3. Secondary reads data from oplog and update them selves.

Mongo oplog diagram

MongoDB : How to write Stored Procedures in MongoDB

The first thing you need to know is that MongoDB stored procedures are written in JavaScript. This feels pretty strange at first, but in actual fact it’s far more satisfying to program in a real language than in any of those vendor-specific SQL-based concoctions. It’s also very powerful: MongoDB uses the same JavaScript runtime as FireFox, so you have pretty much every feature of the language, including closures, Iterators and XML support via E4X.

So in MongoDB, a stored procedure is really just a JavaScript function, which must be stored in a special collection named db.system.js. Imagine a simple function like so:

function addNumbers( x , y ) {
    return x + y;

To convert that to a stored proc, just insert it into db.system.js:

>{_id:"addNumbers", value:function(x, y){ return x + y; }});

Stored procs can be viewed, altered and deleted in the same way as any document in any collection, as that’s all they are. So we can check that our code was successfully stored by using a simple find():

> db.system.js.find()
{ "_id" : "addNumbers", "value" : function cf__3__f_(x, y) {
    return x + y;
} }

Ignore the machine-generated function name. That looks good, and we can start to actually use the proc. That’s done using db.eval():

> db.eval('addNumbers(17, 25)');

That’s all pretty simple, I think.

Python : Dictionary comprehension

Before updating a no sql or mongo doc you need to ensure that you are not adding keys which are not part of initial doc then dictionary comprehension is really helpful.

What it means, is you are able to create a dict (json object) dynamically in single line of code using loops, condition, construction every thing together.

Single line comprehension

items_to_update = {key: value for key, value in payload.items() if key in req_items}


multi line interpretation

if key in req_items:

….for key, value in payload.items():

…….. dictionary[key]=value

Detailed example of created update doc dic before inserting in mongo db

    req_items = ["property1","property2"]
    items_to_update = {key: value for key, value in payload.items() if key in req_items}
    if items_to_update:"Updating device collection with {}".format(items_to_update))
        self.db.update({"filter": filter}, {"$set": items_to_update}, multi=True)"Successfully updated product details")
except PyMongoError as e:
    logger.critical('Pymongo Exception {}'.format(str(e)))
except Exception as e:
    logger.critical('Exception {}'.format(str(e)))



Mongodb – can’t login via -u root -p bitnami

Are you trying to login like this?

mongo admin --username root --password YOUR_PASSWORD

where YOUR_PASSWORD is the one you can see in your AWS system log:

Setting Bitnami application password to 'YOUR_PASSWORD'

If this doesn’t work, you can also try resetting MongoDB password:

MongoDb : Case in-sensative search using pymongo

As we all know Json is case sensitive and so is MongoDB, hence searching same name which got save in different case might not return any result. The solution is to do case in-sensative search.

Below is the example of pymongo

db.users.find_one({'name': re.compile(username, re.IGNORECASE)})