File Uploads
Overview
This tutorial will show you how to upload a file to a Mongoose web server.
Build and run
Follow the Build Tools tutorial to setup your development environment. Then start a command prompt / terminal and enter the following:
git clone https://github.com/cesanta/mongoose
cd mongoose/examples/file-upload-html-form
make
Possible approaches
There are two possible scenarios:
- The file to be uploaded is small - significantly smaller than the amount of free RAM. For example, we're running Mongoose on an embedded Linux system with 64MB of RAM, and uploading some JSON configuration about 1KB in size. In this case, we can use a standard HTML form upload, and receive the whole file in one POST request.
- The file to be uploaded is large - compared to the amount of free RAM, or
significantly more than that. For example, we want to upload a new filesystem
image of size 512KB to an ESP8266 device which has about 30KB of free RAM.
In this case, there is no way to hold the whole file in memory. It
should be handled in small chunks, and Mongoose should receive each
chunk and append it to the written file, until everything is received.
Here we could follow two paths:
- Send a file using a single POST request, passing file content as-is as the POST body - i.e. use binary upload
- Split a file in small chunks (e.g. 2Kb) on a client side, and send each chunk as an individual POST request
Form upload
This method uses a standard HTML upload form, and works when file size is significantly less than free RAM.
- Build and run the form upload example
cd mongoose/examples/file-upload-html-form make clean all
- Go to
http://localhost:8000
in your browser - Click on "Browse...", select a file to upload
- Click on "submit form". You should see a "Thank you!" message
- Check the web server logs:
2021-11-05 02:02:22 2 main.c:22:cb New request to: [/upload], body size: 1991 2021-11-05 02:02:22 2 main.c:28:cb Chunk name: [field1] filename: [] length: 19 bytes 2021-11-05 02:02:22 2 main.c:28:cb Chunk name: [file1] filename: [hello.txt] length: 1698 bytes
Let's see how that worked.
The web server serves static content from the
web_root
folder, which shows an upload form:When a file is selected, and the "submit form" button gets hit, the browser sends a request like this, where every form field is wrapped between "boundary markers":
POST /upload HTTP/1.1 ...other headers... Content-Type: multipart/form-data; boundary=----MyBound123 ------MyBound123 Content-Disposition: form-data; name="field1" type some text here ------MyBound123 Content-Disposition: form-data; name="file1"; filename="hello.txt" Content-Type: text/x-chdr ... CONTENTS OF THE FILE hello.txt ... ------MyBound123--
Mongoose receives that request, fully buffers it in memory, and then iterates over every form field using the mg_http_next_multipart() function:
There, the code simply logs everything. Alternatively, we could save an uploaded file, or write to a flash region, etcetera.
The source code for this example is available at examples/file-upload-html-form
Binary upload, signle POST
When Mongoose receives a large HTTP request, it buffers incoming data, and
for each received chunk of data, it generates MG_EV_HTTP_CHUNK
request. When
a full HTTP message is buffered, then a last, zero-length MG_EV_HTTP_CHUNK
is generated, followed by MG_EV_HTTP_MSG
.
The server side, however, may not wait until the full message is buffered in
memory, but it can delete the incoming chunk using the
mg_http_delete_chunk() function.
If chunks are deleted, then MG_EV_HTTP_MSG
is not generated after the final
zero-length chunk. This way, a server can process chunks as they arrive - for
example, by writing a chunk into a file.
The
MG_EV_HTTP_CHUNK
messages are also generated for the form uploads. However, Mongoose does not strip multipart markers. If form-uploaded message is saved to a file, it'll contain multipart markers.
0-length chunk is the last chunk. Use the MG_IO_SIZE
build constant to limit
the maximum chunk size on the server side.
- Build and run the single POST example
cd mongoose/examples/file-upload-single-post make clean all
- Go to
http://localhost:8000
in your browser - Click on "choose file...", select a file to upload
- You should see a result upload message showing the file size
- Check the web server logs, you should see each chunk details printed:
To have the file contents printed, just remove the comments in the example18149d3d440 2 main.c:17:cb Got chunk len 1448 18149d3d440 2 main.c:18:cb Query string: [name=myfile]
- You can also try using curl:
curl -H Expect: --data-binary @filename http://localhost:8000/upload?name=myfile ok (chunked)
- In particular, we can instruct curl to use chunked transfer encoding and it will send the file in chunks:
In this case, curl used 16372-byte chunks:curl -H Expect: -H "Transfer-Encoding: chunked" --data-binary @filename http://localhost:8000/upload?name=myfile ok (chunked)
18149d445c0 2 main.c:17:cb Got chunk len 16372 18149d445c0 2 main.c:18:cb Query string: [name=myfile]
Let's see how that worked.
The web server serves static content from the
web_root
folder, which shows an upload form:When a file is selected, the browser runs a JS snippet that sends it over to the server:
The server receives each chunk, processes it and deletes it. When the last chunk is received, sends a response.
The source code for this example is available at examples/file-upload-single-post
Binary upload, multiple POSTs
Another way to upload a big file is to split the transfer into several POSTs.
The function mg_http_upload() can parse a POST request with extra information in the query string parameters.
- Build and run the multiple POSTs example
cd mongoose/examples/file-upload-multiple-posts make clean all
- Go to
http://localhost:8000
in your browser - Click on "choose file...", select a file to upload
- You should see a file upload progress indication
- Check your file system, your file should be there
Windows users: the example writes the file to
/tmp
, create aTMP
directory in yourC:
drive
Let's see how that worked.
The web server serves static content from the
web_root
folder, which shows an upload form:When a file is selected, the browser runs a JS snippet that splits the file content and sends it over in several POSTs, each 2KB in length. In the query string, we send the file name and the offset within the file as parameters:
The server receives each POST request and handles it to
Note we remove any possibility for the caller to point to a directory above the one selected, and we also limit the maximum file size to be below 100KB.mg_http_upload()
, that appends the contents to the specified file at the specified offset
The source code for this example is available at examples/file-upload-multiple-posts