Protobuf – Google’s data interchange format which is smaller, faster, and simpler: 1. Getting started with Protobuf using C# runtime library for Protocol Buffers, 2. Handling Dates & Times, 3. Supporting C# Nullable Types, 4. Protobuf Serialization & Deserialization using C#, 5. Converting Protobuf to JSON & JSON to Protobuf

In one of our client projects, we were using JSON as a default message format for data interchange between various distributed services. We were sending out large amount and frequently updated data among our distributed services. So, we were  evaluating other formats (eg. MessagePack, Protobuf, BSON, etc.) to compact message size  & improve the performance. After completing our evaluation, we finalized on Protobuf as our default message format.

Protobuf is the binary format crafted by Google which surpasses JSON performance (i.e. it is smaller, faster, and simpler). It is a language-neutral, platform-neutral, extensible mechanism for serializing structured data.

In this post, I will be explaining how we can quickly get started with Protobuf using C# runtime library for Protocol Buffers with only few steps.

Step 1: Install Google.Protobuf & Google.Protobuf.Tools Nuget Packages

Google.Protobuf: C# runtime library for Protocol Buffers

Google.Protobuf.Tools: Tools for Protocol Buffers

Step 2: Define message formats in a .proto file: message definitions (schema)

For this post, I have created a message format for a typical Order Info class.

// Declares the syntax version being used (Version 3)
syntax = "proto3";

// Specifies the namespace to be used for the generated C# types. 
option csharp_namespace = "SampleApplication.ProtoBuf.Model";

//  Google's "Well Known Types" extensions: DateTime
import "google/protobuf/timestamp.proto"; 

//  Google's "Well Known Types" extensions: Nullable types
import "google/protobuf/wrappers.proto"; 

message OrderInfo {

  message DESTINATION {
    string name = 1;
  }

  DESTINATION destination = 1;

  message ORDERDATA {
    string sourceOrderId = 1;

    message ITEMS {
      string sku = 1;
      string sourceItemId = 2;

      message COMPONENTS {
        string code = 1;
        bool fetch = 2;
        string path = 3;
      }

      repeated COMPONENTS components = 3;
    }

    repeated ITEMS items = 2;

    message SHIPMENTS {

      message SHIPTO {
        string name = 1;
        string companyName = 2;
        string address1 = 3;
        string town = 4;
        string postcode = 5;
        string isoCountry = 6;
      }

      SHIPTO shipTo = 1;

      message CARRIER {
        string code = 1;
        string service = 2;
      }

      CARRIER carrier = 2;
    }

    repeated SHIPMENTS shipments = 3;
    
    // DateTime Type
    google.protobuf.Timestamp submittedDateTime = 4;
    uint32 orderTotal = 5;

    // Nullable Type uint?
    google.protobuf.UInt32Value orderDiscountAmount = 6;
  }

  ORDERDATA orderData = 2;
}

Note:

For supporting Nullable Types:

import "google/protobuf/wrappers.proto";

And then can use it:

google.protobuf.UInt32Value orderDiscountAmount = 6;

For supporting Dates & Times:

import "google/protobuf/timestamp.proto";

And then can use it:

google.protobuf.Timestamp submittedDateTime = 4;

Step 3: Use the protoc protocol buffer compiler to generate C# classes which are needed to read and write messages

protoc -I=$SRC_DIR --csharp_out=$DST_DIR $FILEPATH

Here:

$SRC_DIR = Source directory (where your application’s source code lives)
$DST_DIR = Destination directory (where you want the generated code to go)
$FILEPATH = Path of .proto file

For location of protoc complier:

Generate C# classes from OrderInfo.proto using protoc compliler.

Step 4: Serialization

private static byte[] SerializeOrderInfo(OrderInfo orderInfo)
{
  byte[] protoBufMessageBytes;

  using (var stream = new MemoryStream())
  {
    // Save the Order Info to a stream
    orderInfo.WriteTo(stream);
    protoBufMessageBytes = stream.ToArray();
  }

  return protoBufMessageBytes;
}

Step 4: Deserialization

private static OrderInfo DeserializeOrderInfo(byte[] protoBufMessageBytes) 
            => OrderInfo.Parser.ParseFrom(protoBufMessageBytes);

Step 5: Protobuf to JSON – using JsonFormatter (Reflection-based converter from messages to JSON)

private static string OrderInfoToJson(OrderInfo orderInfo)
{
  var jsonFormatter = JsonFormatter.Default;
  string orderInfoJson = jsonFormatter.Format(orderInfo);
  return orderInfoJson;
}

Step 6: JSON to Protobuf – using JsonParser (Reflection-based converter from JSON to messages)

private static OrderInfo OrderInfoFromJson(string orderInfoJson) 
            => OrderInfo.Parser.ParseJson(orderInfoJson);

Sample Order Info Fill Method:

private static OrderInfo FillOrderInfo()
{
  var orderInfo = new OrderInfo
  {
    Destination = new OrderInfo.Types.DESTINATION
    {
      Name = "accountName"
    },
    OrderData = new OrderInfo.Types.ORDERDATA
    {
      OrderTotal = 98765,
      SubmittedDateTime = Google.Protobuf.WellKnownTypes.Timestamp.FromDateTime(DateTime.UtcNow),
      SourceOrderId = "1234512345",
      OrderDiscountAmount = null,
    }
  };

  var item = new OrderInfo.Types.ORDERDATA.Types.ITEMS
  {
    Sku = "Business Cards",
    SourceItemId = "1234512346",
  };

  var component = new OrderInfo.Types.ORDERDATA.Types.ITEMS.Types.COMPONENTS
  {
    Code = "Content",
    Fetch = true,
    Path = "http://www.example.com/businessCard.pdf"
  };

  item.Components.Add(component);

  orderInfo.OrderData.Items.Add(item);

  var shipments = new OrderInfo.Types.ORDERDATA.Types.SHIPMENTS
  {
    Carrier = new OrderInfo.Types.ORDERDATA.Types.SHIPMENTS.Types.CARRIER
    {
      Code = "fedex",
      Service = "ground"
    },
    ShipTo = new OrderInfo.Types.ORDERDATA.Types.SHIPMENTS.Types.SHIPTO
    {
      Address1 = "1234 Main St.",
      CompanyName = "Acme",
      IsoCountry = "US",
      Name = "John Doe",
      Postcode = "12345",
      Town = "Capitol",
    }
  };

  orderInfo.OrderData.Shipments.Add(shipments);
  return orderInfo;
}

Sample Order Info JSON:

{
  "destination": {
    "name": "accountName"
  },
  "orderData": {
    "sourceOrderId": "1234512345",
    "items": [
      {
        "sku": "Business Cards",
        "sourceItemId": "1234512346",
        "components": [
          {
            "code": "Content",
            "fetch": true,
            "path": "http://www.example.com/businessCard.pdf"
          }
        ]
      }
    ],
    "shipments": [
      {
        "shipTo": {
          "name": "John Doe",
          "companyName": "Acme",
          "address1": "1234 Main St.",
          "town": "Capitol",
          "postcode": "12345",
          "isoCountry": "US"
        },
        "carrier": {
          "code": "fedex",
          "service": "ground"
        }
      }
    ],
    "submittedDateTime": "2021-07-08T11:09:08.727181600Z",
    "orderTotal": 98765
  }
}

That It !!!