My Profile Photo

Stefan Prodan


Co-Founder of Veritech Solutions. Passionate about software architecture, domain driven design and DevOps. Loves programming in .NET, JS and Go


ASP.NET Core vs Go data ingestion benchmark

Two year ago I’ve developed a data ingestion system and I’m planning to migrate it from Windows to Linux and Docker.

This system is composed of two types of services that are communicating over HTTP:

  • a front-end service (Web API) that receives various payloads in JSON format, validates them, does data transformation and aggregation then sends the data to the back-end service
  • a back-end service (Web API) that receives data from the front-end services and based on the payload type it persists it to various storages (PG, ES, Redis, OpenStack Swift)

At peak hours the current system has an ingestion rate of 1000 payloads per second and it’s using about 4GB RAM. Before porting it to a different technology, I have to make sure that the new tech can handle this kind of load and I also want a smaller memory footprint. Since I enjoy writing APIs with C# and Go, I decided to code a basic data flow in both technologies and run some tests on a staging server that has similar specs with the production ones.

Data ingestion prototype

The prototype is composed of two web apps that are running in Docker containers.

Front-end:

  • exposes a HTTP POST endpoint on which it receives a JSON payload from a caller
  • on receive it deserializes the payload into an object
  • serializes the object back to JSON
  • instantiates an HTTP client and makes an HTTP POST request to the back-end service
  • waits for the back-end service to process the payload
  • disposes the HTTP client and returns a HTTP 200 Code to the caller

Back-end:

  • exposes an HTTP POST endpoint on which it receives a JSON payload from the front-end service
  • on receive it deserializes the payload into an object
  • returns a HTTP 200 Code to the front-end service

Both services are instrumented with Prometheus. Prometheus collects the following metrics: rate of HTTP requests per second, HTTP requests latency, CPU, RAM, NET and IO usage of each container.

data flow

The front-end data ingestion handler in ASP.NET Core looks like this:

[HttpPost]
public IActionResult Event([FromBody]Payload payload)
{
    if (!string.IsNullOrEmpty(_settings.ProxyFor))
    {
        using (var client = new HttpClient())
        {
            client.BaseAddress = new Uri(_settings.ProxyFor);
            var content = new StringContent(JsonConvert.SerializeObject(payload), Encoding.UTF8, "application/json");
            var result = client.PostAsync("/ingest/data", content).Result;
            result.RequestMessage.Dispose();
            result.Dispose();
        }
    }

    return new EmptyResult();
}

And the back-end handler:

[HttpPost]
public IActionResult Data([FromBody]Payload payload)
{
    dynamic data = JsonConvert.DeserializeObject<dynamic>(payload.Data);

    return new EmptyResult();
}

The front-end data ingestion handler in Go:

func eventIngestHandler(w http.ResponseWriter, r *http.Request) {
    decoder := json.NewDecoder(r.Body)
    var p Payload
    err := decoder.Decode(&p)
    if err != nil {
        http.Error(w, http.StatusText(400), 400)
    }
    app, _ := r.Context().Value("app").(AppContext)
    if app.Role == "proxy" && app.Endpoints != "" {
        endpoints := strings.Split(app.Endpoints, ",")
        for _, endpoint := range endpoints {
            err := redirectPayload(p, endpoint+"/ingest/data")
            if err != nil {
                http.Error(w, http.StatusText(502), 502)
            }
        }
    }
}

func redirectPayload(p Payload, url string) error {
    b := new(bytes.Buffer)
    json.NewEncoder(b).Encode(p)
    r, err := http.Post(url, "application/json; charset=utf-8", b)
    if err != nil {
        return err
    }
    defer r.Body.Close()
    return nil
}

And the back-end handler in Go:

func dataIngestHandler(w http.ResponseWriter, r *http.Request) {
    decoder := json.NewDecoder(r.Body)
    var p Payload
    err := decoder.Decode(&p)
    if err != nil {
        http.Error(w, http.StatusText(400), 400)
    }
}

Load testing specs

The ASP.NET Core apps are using the microsoft/dotnet latest (1.0.1) Docker image and the Go apps are using golang:1.7.1-alpine image.

Server specs:

  • Ubuntu 16.04.1 LTS x64
  • Docker 1.12.2
  • Intel Xeon X5650 12M Cache, 2.66 GHz, 12 threads
  • 12 GB RAM
  • Intel 82575EB Gigabit Ethernet Controller

Load test machine specs:

  • Ubuntu 16.04.1 LTS x64
  • ApacheBench v2.3
  • Intel Core i7-4790 8M Cache, 3.60 Ghz, 8 threads

ApacheBench command:

 ab -k -l -p payload.json -T application/json -c 50 -n 10000 http://<server-ip>:<app-port>/ingest/event

JSON payload:

{
"data": "{'job_id':'c4bb6d130003','container_id':'ab7b85dcac72','status':'Success: process exited with code 0.'}"
}

Load testing results

ApacheBench reported that the ASP.NET Core front-end service has processed 10K request in 31 seconds at a rate of 319 request per second:

Concurrency Level:      50
Time taken for tests:   31.348 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1160000 bytes
Total body sent:        2950000
HTML transferred:       0 bytes
Requests per second:    319.00 [#/sec] (mean)
Time per request:       156.741 [ms] (mean)
Time per request:       3.135 [ms] (mean, across all concurrent requests)
Transfer rate:          36.14 [Kbytes/sec] received
                        91.90 kb/s sent
                        128.04 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0      34
Processing:     8  156 175.9     98    1144
Waiting:        8  156 175.9     98    1144
Total:          8  156 175.9     98    1144

ApacheBench reported that the Go front-end service has processed 10K request in 3 seconds at a rate of 3213 request per second:

Concurrency Level:      50
Time taken for tests:   3.112 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1400000 bytes
Total body sent:        2950000
HTML transferred:       0 bytes
Requests per second:    3213.09 [#/sec] (mean)
Time per request:       15.561 [ms] (mean)
Time per request:       0.311 [ms] (mean, across all concurrent requests)
Transfer rate:          439.29 [Kbytes/sec] received
                        925.65 kb/s sent
                        1364.94 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.7      0      33
Processing:     4   15  15.9     13     380
Waiting:        4   15  15.9     13     380
Total:          4   15  16.3     13     385

Prometheus reported the following resource usage:

Service	           CPU     RAM       RAM (after test)
-----------------------------------------------------
ASP.NET front-end  12.65%  224.04MB  104.74MB
ASP.NET back-end   4.23%   134.15MB  54.98MB
Go front-end       10.95%  9.13MB    4.40MB
Go back-end        3.86%   6.72MB    3.36MB

Improving ASP.NET Core HTTP client code

Instead of creating a HTTP client on each call I changed the code and used a static client and switched to async:

private static HttpClient client = new HttpClient();

[HttpPost]
public async Task<IActionResult> Event([FromBody]Payload payload)
{
    if (!string.IsNullOrEmpty(_settings.ProxyFor))
    {
        var content = new StringContent(JsonConvert.SerializeObject(payload), Encoding.UTF8, "application/json");
        var result = await client.PostAsync(_settings.ProxyFor + "/ingest/data", content);
        result.RequestMessage.Dispose();
        result.Dispose();
    }
    return new EmptyResult();
}

The results improved a lot. ApacheBench reported that the ASP.NET Core front-end service has processed 10K request in 10 seconds at a rate of 936 request per second:

Concurrency Level:      50
Time taken for tests:   10.674 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1160000 bytes
Total body sent:        2950000
HTML transferred:       0 bytes
Requests per second:    936.87 [#/sec] (mean)
Time per request:       53.369 [ms] (mean)
Time per request:       1.067 [ms] (mean, across all concurrent requests)
Transfer rate:          106.13 [Kbytes/sec] received
                        269.90 kb/s sent
                        376.03 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0      10
Processing:     5   53  11.3     51     235
Waiting:        5   53  11.3     51     235
Total:          5   53  11.4     51     239

Scale up with Docker Swarm

My goal is to make the system take 1K req/sec, so I decided to run the containers on Docker Swarm and scale the front-end service to x3.

ApacheBench reported that the ASP.NET Core front-end service x3 has processed 10K request in 6 seconds at a rate of 1615 request per second:

Concurrency Level:      100
Time taken for tests:   6.190 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1160000 bytes
Total body sent:        2950000
HTML transferred:       0 bytes
Requests per second:    1615.48 [#/sec] (mean)
Time per request:       61.901 [ms] (mean)
Time per request:       0.619 [ms] (mean, across all concurrent requests)
Transfer rate:          183.00 [Kbytes/sec] received
                        465.40 kb/s sent
                        648.40 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0      16
Processing:     5   60  28.2     53     415
Waiting:        5   59  28.2     53     415
Total:          5   60  28.4     53     419

As expected, Docker Swarm balanced the load between the front-end instances and I could reach over 1K req/sec. The back-end service didn’t need to be scaled at this load. I did a load test on the back-end service (single instance) and it got up to 4K req/sec.

For the Go app I’ve raised the stakes to 100K requests at a concurrency level of 100. ApacheBench reported that the Go front-end service x3 has processed 100K request in 11 seconds at a rate of 9017 request per second:

Concurrency Level:      100
Time taken for tests:   11.089 seconds
Complete requests:      100000
Failed requests:        0
Keep-Alive requests:    100000
Total transferred:      14000000 bytes
Total body sent:        29400000
HTML transferred:       0 bytes
Requests per second:    9017.95 [#/sec] (mean)
Time per request:       11.089 [ms] (mean)
Time per request:       0.111 [ms] (mean, across all concurrent requests)
Transfer rate:          1232.92 [Kbytes/sec] received
                        2589.14 kb/s sent
                        3822.06 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0      22
Processing:     4   11   7.7     10     430
Waiting:        4   11   7.7     10     430
Total:          4   11   7.8     10     435

Observations

By default the ASP.NET Core garbage collector System.GC.Server mode is enabled, this works well on Windows but on Linux made the front-end service go up to 2GB of memory. After it reached 2GB, the HTTP client started to crash on every request. If you are deploying ASP.NET Core with Docker then you should set "System.GC.Server": false in project.json.

I’ve run this test for 6 hours every 10 minutes, while the Go performance was consistent on every run, the ASP.NET Core fluctuated a lot. I’ve seen ASP.NET Core jump to 20% CPU and drop to 200 req/s some times.

When I’ve run the tests with 100K requests at a concurrency level of 100, both HTTP clients crashed after a while. In Golang I could fix this by disabling KeepAlive and setting the MaxIdleConnsPerHost to 100. I suppose there is a way to set the same things on ASP.NET Core too.

Conclusions

Benchmark results:

     	        REQ/sec   TIME     REQ    Concurrency  Memory
-------------------------------------------------------------
ASP.NET Core    936       10sec    10K    50           224MB
Go              3213      3sec     10K    50           9MB
ASP.NET Core    1324      75sec    100K   300          235MB
Go              6051      16sec    100K   300          12MB

For my use case, the load tests showed that the Go HTTP stack is way faster then ASP.NET Core. Scaling the front-end service to x3 made the ASP.NET Core reach my 1K req/s goal, but the memory usage is very high compared to Go.

I know that ASP.NET Core is for all intents and purposes a brand new platform. However, running the load test on the back-end service resulted in 4K req/s which leads me to believe that, while Kestrel is very fast, there may be a bottleneck when sending HTTP requests. The .NET Core team has made some huge improvements on the Kestrel server performance this year. In terms of serving data, Kestrel has hit 1M req/sec, so I expect the data ingestion rate to improve in the future.

If you wish to run the tests yourself, the ASP.NET Core repo is on GitHub here and the Go repo here.

comments powered by Disqus