How to Write to a File in C

File input/output (I/O) is a fundamental capability required for many C programs, yet doing it efficiently and robustly requires deeper understanding of how file writing works under the hood.

In this comprehensive guide, we will build on the basic file output concepts in C to explore writing performance, safety, portability, and the best practices for building high-performance applications.

File I/O Basics and Beyond

We start with a quick review of basic file writing in C using the standard library:

// Open file
FILE *fp = fopen("/path/to/file", "w"); 

// Write to file
fprintf(fp, "Hello world!");

// Close file 
fclose(fp);

Underneath these simple commands lies a complex interaction with your operating system‘s virtual file system and actual physical media. There are also limitations of disks and networks to consider as they relate to file write speeds.

We dive into these concepts next…

Buffered vs Unbuffered Writes

By default, C employs buffered file writing for performance. This means data is held in an in-memory buffer and only flushed to disk intermittently by the OS based on buffer size and file type.

Unbuffered writing guarantees data written to physical media synchronously before functions return, but at a performance cost:

// Unbuffered write
setvbuf(fp, NULL, _IONBF, 0);  

fwrite(buffer, size, count, fp); // Blocks til written

Compare to buffered writing, where fwrite() returns quickly without waiting for physical I/O completion.

Tradeoffs involve throughput vs consistency guarantees. For example, buffered writing maximizes speed writing to log files where losing recent entries on crash may be acceptable.

Managing Buffer Size

applications and resources. Or explicitly via setvbuf():

For block buffered files, buffer size greatly impacts throughput and kernel overhead trading off:

• Larger buffers reduce OS round trips improving throughput
• But consume more memory
• And increase data loss risk on failures

Apps writing formatted data like JSON may bound buffer size to limit partial data corruption. Or when writing critical data, bounds prevent huge buffered data loss on crash:

// Bound write buffer size 
setvbuf(fp, buf, _IOFBF, 1024); 

fwrite(my_data);

When to Flush

In addition to buffer size, flushing strategy can significantly impact reliability/speed.

While the standard library handles flushing output buffers on fclose() and file repositioning, explicitly flushing at key points may be necessary. This forces synchronization to physical media ensuring critical data writes.

For example, flushing application state files on updates:

fprintf(fp, "Update complete!");

// Ensure write before continuing 
fflush(fp); 
fdatasync(fileno(fp));

next_task();

Using fflush() followed by fdatasync() ensures both stream buffer and OS cached data reaches disk before proceeding.

Sequential vs Random Writes

Due to the physical properties of magnetic media, sequential writes substantially outperform random access writes in throughput, simplified:

Transfer Type	Throughput	Latency
Sequential Write	Up to 200MB/s	20 ms
Random Write	~1 MB/s	10-12 ms

Therefore, restructuring buffered output to be sequential where possible is key for performance on rotational disks.

SSDs help by providing <100x lower random write latency, but sequentials still win in throughput.

The Cost of File System Metadata

Beyond raw media speeds, remember that each file write transaction requires updating inode metadata bitmap and indices – an expensive operation!

File systems optimize by batching metadata changes only syncing periodically. But under load Metadata bottlenecking writes. Plan buffer strategy accordingly.

There are many more esoteric considerations like fragmentation, journaling mechanisms, RAID parity tradeoffs. But knowledge of fundamentals will optimize most apps!

Going Parallel: Concurrency and Thread Safety

Taking advantage of multi-core systems requires concurrent execution. But shared file handles require thread safety mechanisms.

Let‘s explore issues around threading file output…

Locking Shared File Pointers

By default, the stdio library manages internal buffer locking for simple cases. But direct use of POSIX read()/write() require external locking:

// Thread safe write via shared fp

pthread_mutex_lock(&fp_lock); 

fwrite(buffer, size, count, fp);

pthread_mutex_unlock(&fp_lock);

Failure to lock shared file pointers risks interleaved/corrupted data.

Producer/Consumer Buffering

More advanced thread safety structures like producer/consumer circular buffers allow concurrent lock-free buffering:

         +-------+   
Producer >| Buffer |> File
         +-------+

             ^
             |

Consumer    |

This enables fast single-thread buffered writes decoupled from compression, encryption or disk I/O workers operating on buffers.

Per-Thread Buffers

Or avoid locks entirely through per-thread personal buffers written by workers then collected and flushed sequentially to file by master thread:

Thread 1 > Buffer 1 > File
Thread 2 > Buffer 2 > 
Thread 3 > Buffer 3 >

Trading memory for performance by avoiding shared state access.

Portability Considerations

While POSIX defines common open()/write()/close() interface, portability considerations remain around buffering, atomicity guarantees, available features.

For example Linux offers enhanced fwrite_unlocked() optimized for single threaded writes. But non-portable.

Or precise data flushing requires OS-specific interfaces:

• Linux/BSD: sync_file_range()
• Windows: FlushFileBuffers()
• Older Unix: fsync()

Understanding compatibility tradeoffs allows adapting high-performance patterns across environments.

Putting it All Together: Best Practices

We‘ve covered alot of territory! Here are some key takeaways for performant portable file writing:

• Leverage sequential writes over random access where possible
• Size buffers based on expected data sizes
• Flush explicitly after critical data points
• Consider thread safety mechanisms like locking, separate buffers
• Understand OS/FS specific bottlenecks like metadata updates

Beyond concrete recommendations, profile output patterns on target deployment systems! Vary buffer sizes, parallelism, flush frequency and measure throughput and loss behaviors.

Only real-world behavioral analysis provides system specific optimizations.

So get out there measuring and refining your buffered file writing approach!

References

• [1] Linux System Calls Handbook http://man7.org/linux/man-pages/man2/write.2.html
• [2] Beej‘s Guide to File I/O http://beej.us/guide/bgipc/output/html/multipage/getFile.html
• [3] Wikipedia OS Page Cache https://en.wikipedia.org/wiki/Page_cache

About the Author

I am a Linux system engineer with over a decade optimizing applications for file writing throughput. Please reach out with any questions!

john@example.com