MongoDB : History tracking micro service

Mongo DB local -> oplog.$main collection keeps records of all changes which happen in the primary database, which are eventually read by secondaries to catch up data or syncup data in the mongo replica set.

A micro service can be written, which can pull up data / delta form oplog.$main based on the name space ( defined db and collection) and can save that data in destination Db or audit DB.

Later on audit DB can the queried to fetch the history or log data pertaining to entity.

audit log sequence diagram

Advertisements

Architecture: How mongodb replication works and secondary node get data from primary node

Here is the simple diagram which explains how the middle man oplog watches/monitors primary and pull any new operation or changes happened on primary and records them in oplog, which eventually get read by all secondaries to update them selves.

  1. insert operation happened in primary
  2. Operation got logged in oplog ( which is part of local db – oplog.$main )
  3. Secondary reads data from oplog and update them selves.

Mongo oplog diagram

Transaction ACID properties

A database transaction is a sequence of actions that are treated as a single unit of work. These actions should either complete entirely or take no effect at all. Transaction management is an important part of RDBMS-oriented enterprise application to ensure data integrity and consistency. The concept of transactions can be described with the following four key properties described as ACID −

  • Atomicity − A transaction should be treated as a single unit of operation, which means either the entire sequence of operations is successful or unsuccessful.
  • Consistency − This represents the consistency of the referential integrity of the database, unique primary keys in tables, etc.
  • Isolation − There may be many transaction processing with the same data set at the same time. Each transaction should be isolated from others to prevent data corruption.
  • Durability − Once a transaction has completed, the results of this transaction have to be made permanent and cannot be erased from the database due to system failure.

MongoDB : How to write Stored Procedures in MongoDB

The first thing you need to know is that MongoDB stored procedures are written in JavaScript. This feels pretty strange at first, but in actual fact it’s far more satisfying to program in a real language than in any of those vendor-specific SQL-based concoctions. It’s also very powerful: MongoDB uses the same JavaScript runtime as FireFox, so you have pretty much every feature of the language, including closures, Iterators and XML support via E4X.

So in MongoDB, a stored procedure is really just a JavaScript function, which must be stored in a special collection named db.system.js. Imagine a simple function like so:

function addNumbers( x , y ) {
    return x + y;
}

To convert that to a stored proc, just insert it into db.system.js:

> db.system.js.save({_id:"addNumbers", value:function(x, y){ return x + y; }});

Stored procs can be viewed, altered and deleted in the same way as any document in any collection, as that’s all they are. So we can check that our code was successfully stored by using a simple find():

> db.system.js.find()
{ "_id" : "addNumbers", "value" : function cf__3__f_(x, y) {
    return x + y;
} }

Ignore the machine-generated function name. That looks good, and we can start to actually use the proc. That’s done using db.eval():

> db.eval('addNumbers(17, 25)');
42

That’s all pretty simple, I think.

Python : Dictionary comprehension

Before updating a no sql or mongo doc you need to ensure that you are not adding keys which are not part of initial doc then dictionary comprehension is really helpful.

What it means, is you are able to create a dict (json object) dynamically in single line of code using loops, condition, construction every thing together.

Single line comprehension

items_to_update = {key: value for key, value in payload.items() if key in req_items}

 

multi line interpretation

if key in req_items:

….for key, value in payload.items():

…….. dictionary[key]=value

Detailed example of created update doc dic before inserting in mongo db

try:
    req_items = ["property1","property2"]
    items_to_update = {key: value for key, value in payload.items() if key in req_items}
  
    if items_to_update:
        logger.info("Updating device collection with {}".format(items_to_update))
        self.db.update({"filter": filter}, {"$set": items_to_update}, multi=True)
        logger.info("Successfully updated product details")
except PyMongoError as e:
    logger.critical('Pymongo Exception {}'.format(str(e)))
except Exception as e:
    logger.critical('Exception {}'.format(str(e)))

 

asd

Mongodb – can’t login via -u root -p bitnami

Are you trying to login like this?

mongo admin --username root --password YOUR_PASSWORD

where YOUR_PASSWORD is the one you can see in your AWS system log:

Setting Bitnami application password to 'YOUR_PASSWORD'

If this doesn’t work, you can also try resetting MongoDB password:

http://wiki.bitnami.com/Components/mongoDB?highlight=mongo#How_to_reset_the_MongoDB_root_password.3f