Huge Response

Overview

This tutorial will show you how to send large amounts of data, larger than available buffer memory. We'll do that with an HTTP server but the same technique could be applied to your own protocol.

On devices with limited RAM, it is important to limit the response size of API calls. The concept here is to fetch a large amount of data in smaller chunks, and guarantee its integrity. Data gets returned in a series of request/response transactions, where each response is small enough to fit into the available device RAM.

Data integrity is implemented by versioning. The idea is that the first response includes the current "version" of the data, and that version is passed to all subsequent requests. If the data gets modified, then its version will change; if this happens in the middle of the series of requests, the client will request an outdated version and so the server will respond with a "wrong version" error. In real life, we'll repeat this procedure as needed to get the whole data.

In this example, the version gets updated every 5 seconds, so try several times.

Build and run

• Follow the Build Tools tutorial to setup your development environment.
• Start a terminal in the project directory; if you've not already done so, clone the Mongoose Library repo

  git clone https://github.com/cesanta/mongoose
  

• Build and run the example

  cd mongoose/tutorials/http/huge-response
  make clean all
  

• Go to http://localhost:8000 in your browser
• Click on "fetch data" in your browser; you'll either see all the data and its current version, or an error indication

If you want to see the inner workings of this example, compile with debug logging enabled:

int main(void) {
  struct mg_mgr mgr;
  mg_log_set(MG_LL_DEBUG);  // Add this line
  mg_mgr_init(&mgr);

• Then observe the log, you may see
  • the transfer upto a version change, with an error indication

    39ea985 2 main.c:54:fn                  {"start":0,"version":0}
    39ea985 2 main.c:67:fn                  {"version":1,"start":0,"data":[1979305336,367846725,...]}
    ...
    39eaf82 2 main.c:51:fn                  {"start":8600,"version":1}
    39eaf82 2 main.c:63:fn                  {"error":"wrong version", "version":1}
    

  • the whole transfer

    3a063cb 2 main.c:51:fn                  {"start":0,"version":0}
    3a063cb 2 main.c:63:fn                  {"version":24,"start":0,"data":[1164495442,188059193,...]}
    ...
    3a069b7 2 main.c:51:fn                  {"start":10000,"version":24}
    3a069b7 2 main.c:63:fn                  {"version":24,"start":10000,"data":[]}
    

How it works

Server side

We'll concentrate here on those aspects related to sending huge responses, for the details of initializing and setting up an HTTP server please check the HTTP server tutorial.

When we get an HTTP request for the /api/data URI, we extract the start and version parameters from the JSON object. If the requested version is the same as our current version, we return a JSON object containing the data; otherwise we return an error indication.

The function getparam() makes use of the function mg_json_get_num(), that parses a JSON string in a struct mg_str, searching for a valid numeric value at the JSON path json_path. If successful, it will store its value at placeholder dv and return true; returning false otherwise.

Take a look at the JSON-RPC tutorial for more details on how to use Mongoose's built-in JSON API.

To craft these JSON messages, we take advantage of the function mg_http_reply(). It is similar to printf(), and here we use two convenient extensions to the syntax.

• The specifier M is used to call a function, passing it the remaining parameter list. This function must return the number of bytes it has printed
• The specifier m does the same, but also double-quotes what it prints. Here we use it to print JSON-escaped strings by calling mg_print_esc().

Using a function call inside the print function allows us not only to have a clean code showing the message being sent, and handle the details of how much data is sent on a different function, but also set aside the hassle of determining how much memory allocate for that task, thing that mg_http_reply() handles under the hood for us.

The function being called will receive an output function pointer and its parameter, plus a variable argument list containing the remaining parameters; in our case start, the start address.

Inside this function, we'll use mg_xprintf(). It takes the very same parameters we received, prints where it has been told to do it, and returns how many bytes it printed. We determine if the remaining data length is larger than our chunk size, and print the smallest amount of members of the data array.

#define DATA_SIZE 10000        // Total number of elements
#define CHUNK_SIZE 100         // Max number returned in one API call
static int s_data[DATA_SIZE];  // Simulate some complex big data

Client side

The JavaScript code in the browser contains a function that POSTs a JSON object to /api/data requesting the start address and the version for the desired data; recursively calling itself on successful transactions that contain some data.

On a "fetch data" button click, this function gets called with offsetset to zero, and version is initialized to zero, so the server will send us the current version and start at zero.

Every time we receive a chunk, we request another one starting at the next start address (that is, we add the amount of bytes we received), and specify the current version. As long as the data version stays current, we'll get another chunk. If the data version has changed, we'll get an error indication. When all data has been transferred, we'll get an empty array, that is, with zero length. In both cases, the function load() will update the screen and exit.

Browse latest code