Google Launcher vs. Google Container Engine MongoDB benchmark

We have tested two MongoDB cluster solutions deployed on Google Cloud Platform: Google Launcher's MongoDB cluster and our MongoDB cluster Docker container running on Google Container Engine. We wanted to simply compare the performance of both solutions using a standard mongo performance benchmark. What are the benchmark results?

Mongo Database benchmark

The workload compares highly available instances of Google Launcher's MongoDB cluster and our own deployment of MongoDB cluster docker image on Google Container Engine. The replication technology used is a standard MongoDB replica set with primary and secondary nodes and maximum of one arbiter node if needed (to have an odd number of nodes in total).

We chose the same smaller instances of 2 nodes + 1 really small Arbiter node for both mongo clusters. We picked the same storage size and type. Google doesn't charge extra for using Google Launcher's deployments and MongoDB as a software is also for free, so you end up paying the exact same price as you would for buying the cloud computational power from Google Compute's services directly. For running MongoDB in Docker containers on Google Container Engine (GKE) you have to pay a bit more extra for the Kubernetes cluster service but it doesn't really matter as the only reason to use this solution is when you are already using GKE in "production".

To determine throughput, a mongo-perf framework was used to generate workload from a mongo client in the same zone as the primary database instance. The workload is a set of step load tests that double the number of threads (connections) with each run. The benchmark's set of tests is a selection of insert and update simple and complex test cases from the standard test package.

Operations per second

Operations per second results show that Google Launcher's MongoDB cluster is faster than our MongoDB Docker cluster on GKE in most of the test cases. We chose three representative tests to show the results.

MMS sets of tests were the only tests with stable results where Google Launcher performed worse than our MongoDB Docker. You can also see how the performace goes rapidly down with more threads used due to synchronization overhead.

The third test Update.DocValidation.OneNum shouldn't cause any synchronization lag between concurrent threads so it shows the replication lag of both MongoDB cluster instances. Google Launcher wins here again.

Running the benchmark

We used the following environment configuration mongo-perf parameters for our testing.

Test environment:

• Google Launcher's MongoDB - 2x n1-highmem-2 (2cpu, 13GB RAM) + f1.micro (shared cpu, 0.6 GB RAM), MongoDB 3.4.1, 2x 150 GB SSD - Read/Write 4500 random IOPS, RW throughput of 72 MB/s
• Our MongoDB Docker on GKE -2x n1-highmem-2 (2cpu, 13GB RAM) + f1.micro (shared cpu, 0.6 GB RAM), MongoDB 3.4.6, 2x 150 GB SSD - Read/Write 4500 random IOPS, RW throughput of 72 MB/s

Test overview:

python benchrun.py --host "$host" --replset "$rs" -u "$user" -p "$pw" -f testcases/* --includeFilter insert update -t 1 2 4 8 16 32 64 128 --out "$output"

75 different insert and update tests were run with this command. 256 threads was more than both instances could handle to serve so the final benchmark was executed for a maximum of 128 threads. The output is saved in JSON format to a file.

The output JSON file is then parsed in a bash script called parse.sh using jq and saved into a CSV file. The CSV files are then automatically processed and used for plotting using gnuplot software.

# for all test cases  
for testName in $( jq -r --sort-keys '.results[].name' "$file" );  
do  
  ...  
  # take the operations per second values and save it to an intermediary output file  
  jq --sort-keys '.results[] | select(.name=='$testName') | .results["1","2","4","8","16","32","64","128"].ops_per_sec' "$file" >$outFile  
  ...  
  # plot the data with gnuplot  
  gnuplot "${testName}.gp"  
done

The benchmark scripts, tests, environment setting and provisioning, the parser, the gnuplot template and more you can find in this github repository.

Result

Google Launcher's MongoDB cluster outperformed our MongoDB Docker cluster on Google Container Engine in most of the test cases of the used benchmark. Google Launcher's MongoDB is probably faster due to no overhead of Docker/Kubernetes and being optimized by Google's Engineers to run on Google Compute VM's.