added fwrite and fread vignette #7216

venom1204 · 2025-07-27T09:26:54Z

closes #2855

This PR adds a vignette on data I/O with fread() and fwrite, addressing issue #2855.

The fread() section includes examples and tips taken from the convinience features of fread.

Covers key options, performance tips, and optimizations (e.g., multi-threading, column selection).

The fwrite() section documents:

Intelligent quoting (quote="auto").
Fine-grained date/time serialization (dateTimeAs).
Support for bit64::integer64.
Column order and subset control.

Includes runnable code snippets for common workflows.

hi @MichaelChirico , @jangorecki , @tdhock can you please have a look when you got time .

thanks ...

codecov · 2025-07-27T09:34:39Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.79%. Comparing base (053d905) to head (d1cb56a).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #7216   +/-   ##
=======================================
  Coverage   98.79%   98.79%           
=======================================
  Files          81       81           
  Lines       15254    15254           
=======================================
  Hits        15070    15070           
  Misses        184      184

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

aitap · 2025-07-27T10:02:35Z

vignettes/datatable-fread-and-fwrite.Rmd

+
+#### 1.1.2 **Reading from R connections and URLs**
+
+`fread()` is highly versatile and can accept R connection objects as input to its file (or input) argument. This allows you to read from various sources, including:


Where was this information sourced?

AFAIK it does not support connections

I've updated the vignette to remove the incorrect statement about R connection support.
The section now only documents reading from file paths and URLs (as character strings)

jangorecki · 2025-07-27T10:04:54Z

I think we rather prefer "fread and fwrite" than "fread() and fwrite()"

jangorecki · 2025-07-27T10:07:12Z

vignettes/datatable-fread-and-fwrite.Rmd

@@ -0,0 +1,339 @@
+---
+title: "fread() and fwrite()"


fread and fwrite
Or
Fast read and Fast write

jangorecki · 2025-07-27T10:08:47Z

vignettes/datatable-fread-and-fwrite.Rmd

+
+Look at this [example](https://stackoverflow.com/questions/36256706/fread-together-with-grepl/36270543#36270543) for more detail.
+
+On Windows we recommend [Cygwin](https://www.cygwin.com/) (run one .exe to install) which includes the command line tools such as grep. In March 2016, Microsoft [announced](https://www.hanselman.com/blog/developers-can-run-bash-shell-and-usermode-ubuntu-linux-binaries-on-windows-10) they will include these tools in Windows 10 natively. On Linux and macOS, these tools have always been included in the operating system. You can find many examples and tutorials about command line tools online. We recommend [Data Science at the Command Line](https://www.oreilly.com/library/view/data-science-at/9781491947845/).


Not sure if vignette is good place for recommending a book, especially if it is not freely available

jangorecki · 2025-07-27T10:09:36Z

vignettes/datatable-fread-and-fwrite.Rmd

+```{r}
+# 1. Create a sample data.table and write it to a gzipped CSV
+set.seed(123)
+original_dt <- data.table(A = 1:5, B = runif(5))


Why not = as assignment operator?

i will update it

aitap · 2025-07-27T13:32:43Z

vignettes/datatable-fread-and-fwrite.Rmd

+output: rmarkdown::html_vignette # <--- Changed
+vignette: >
+  %\VignetteIndexEntry{Fast Read and Fast Write}
+  %\VignetteEngine{knitr::rmarkdown} # <--- Changed


Could you please use the same vignette engine and output format as the rest of the vignettes? Whom are these comments for?

aitap · 2025-07-27T13:37:31Z

vignettes/datatable-fread-and-fwrite.Rmd

+
+Look at this [example](https://stackoverflow.com/questions/36256706/fread-together-with-grepl/36270543#36270543) for more detail.
+
+On Windows we recommend [Cygwin](https://www.cygwin.com/) (run one .exe to install) which includes the command line tools such as grep. In March 2016, Microsoft [announced](https://www.hanselman.com/blog/developers-can-run-bash-shell-and-usermode-ubuntu-linux-binaries-on-windows-10) they will include these tools in Windows 10 natively. On Linux and macOS, these tools have always been included in the operating system. You can find many examples and tutorials about command line tools online.


Modern versions of Rtools consist of Cygwin (well, MSYS2) plus pre-compiled third-party dependencies. In particular, they do contain grep and gawk, which could be enough to let this vignette build on Windows. (But make sure that the awk example works with a POSIX, non-GNU awk.)

aitap · 2025-07-27T13:38:34Z

vignettes/datatable-fread-and-fwrite.Rmd

+all_lines = readLines("example_data.txt")
+data_lines = grep("HEADER", all_lines, value = TRUE, invert = TRUE)
+fread(text = data_lines)


Did the previous revision of this vignette fail to build for you? Now this code is in conflict with the text below because grep -v is nowhere to be seen.

vignettes/datatable-fread-and-fwrite.Rmd

aitap · 2025-07-27T13:45:31Z

vignettes/datatable-fread-and-fwrite.Rmd

+
+`fread()` doesn't directly support SQL `INSERT` scripts, but they can be processed via command-line tools. For example, given `insert_script.sql`:
+
+```{r, eval=FALSE}


Why make this a commented-out R code block? If you can't get SQL syntax highlighting to work (shouldn't it be just ```SQL?), try a non-typed code block delimited by just ```.

aitap · 2025-07-27T13:46:08Z

vignettes/datatable-fread-and-fwrite.Rmd

+file.remove("insert_script.sql")
+```
+
+- The `awk` command transforms each INSERT line into a comma-separated list of its values.


This line conflicts with the code block above.

aitap · 2025-07-27T13:47:33Z

vignettes/datatable-fread-and-fwrite.Rmd

+
+### 1.6 **`integer64` Support**
+
+By default, `fread` detects integers larger than 2³¹ and reads them as `bit64::integer64` to preserve full precision. This behavior can be overridden in three ways:


Markdown is a superset of HTML. Does 2<sup>31</sup> work? Does $2^{31}$ work with our vignette engines? (They don't necessarily work with equations, but <sup> should work.)

both of them are working

vignettes/datatable-fread-and-fwrite.Rmd

…o issue2855

vignettes/datatable-fread-and-fwrite.Rmd

ChristianWia · 2025-07-30T21:04:46Z

welcome for further vignette translation.

vignettes/datatable-fread-and-fwrite.Rmd

ben-schwen · 2025-08-04T09:22:42Z

vignettes/datatable-fread-and-fwrite.Rmd

+
+Look at this [example](https://stackoverflow.com/questions/36256706/fread-together-with-grepl/36270543#36270543) for more detail.
+
+On Windows we recommend [Cygwin](https://www.cygwin.com/) (run one .exe to install) which includes the command line tools such as grep. In March 2016, Microsoft [announced](https://www.hanselman.com/blog/developers-can-run-bash-shell-and-usermode-ubuntu-linux-binaries-on-windows-10) they will include these tools in Windows 10 natively. On Linux and macOS, these tools have always been included in the operating system. You can find many examples and tutorials about command line tools online. We recommend [Data Science at the Command Line](https://www.oreilly.com/library/view/data-science-at/9781491947845/).


Also not sure that we should recommend Cygwin anymore. I'm working on WSL here and its a charm

this is for fread to run commands on windows right?
do you need any special environment to give fread access to unix shell commands from WSL or Rtools?

if so it would be helpful to document

Co-authored-by: Benjamin Schwendinger <52290390+ben-schwen@users.noreply.github.com>

ben-schwen · 2025-08-05T08:20:17Z

vignettes/datatable-fread-and-fwrite.Rmd

+
+When data is written as strings (either inherently, like character columns, or by choice, like `dateTimeAs="ISO"`), `quote="auto"` (default) intelligently quotes fields:
+
+**Contextual Quoting**: Only quotes fields if they contain the delimiter (`sep`), a double quote, or a newline. This ensures clean, RFC 4180-compliant CSVs.


we also quote at carriage return \r or empty strings "" to distinguish from NA! so we are not strictly RFC 4180-compliant

ben-schwen · 2025-08-05T10:20:28Z

vignettes/datatable-fread-and-fwrite.Rmd

+  text_field = c("Contains,a,comma", "Contains \"a quote\"", "Clean_text", "", NA),
+  numeric_field = 1:5
+)
+temp_quote_adv <- tempfile(fileext = ".csv")


why going back to use <- for assignment?

ben-schwen · 2025-08-05T10:21:06Z

vignettes/datatable-fread-and-fwrite.Rmd

+.old.th = setDTthreads(1)
+```
+
+The `fread()` and `fwrite()` functions in the `data.table` R package are not only optimized for speed on large files, but also offer powerful and convenient features for working with small datasets. This vignette highlights their usability, flexibility, and performance for efficient data import and export.


do we really highlight performance?

Should I remove the word from the introduction, or would it be better to add a short new section at the end? The new section could include a simple system.time() comparison of fread/fwrite vs read.csv/write.csv on a 1-million-row table.

ben-schwen

From my perspective the only open thing is the recommendation of the command line book and cygwin as pointed out by Jan.

ben-schwen

LGTM, not sure though if we want to do benchmarks though

tdhock · 2025-08-06T12:32:20Z

you could link atime benchmarks https://tdhock.github.io/blog/2024/pandas-dt/ and https://tdhock.github.io/blog/2023/dt-atime-figures/

tdhock · 2025-08-14T13:22:59Z

great thanks

added fwrite and fread vignette

872dc8b

venom1204 requested a review from MichaelChirico as a code owner July 27, 2025 09:26

aitap reviewed Jul 27, 2025

View reviewed changes

..

66a5ea8

jangorecki reviewed Jul 27, 2025

View reviewed changes

venom1204 added 3 commits July 27, 2025 10:51

<- by =

ef4aa63

..

e468e20

titel change

8fb7ed6

aitap reviewed Jul 27, 2025

View reviewed changes

updated

70a8992

venom1204 requested a review from aitap July 27, 2025 19:05

ben-schwen reviewed Jul 28, 2025

View reviewed changes

vignettes/datatable-fread-and-fwrite.Rmd Outdated Show resolved Hide resolved

ben-schwen reviewed Jul 28, 2025

View reviewed changes

vignettes/datatable-fread-and-fwrite.Rmd Outdated Show resolved Hide resolved

ben-schwen reviewed Jul 28, 2025

View reviewed changes

vignettes/datatable-fread-and-fwrite.Rmd Outdated Show resolved Hide resolved

venom1204 added 2 commits July 30, 2025 06:08

Merge branch 'master' of https://github.com/Rdatatable/data.table int…

fbb7c86

…o issue2855

added changes suggested by ben-schwen

77f26c4

ben-schwen reviewed Jul 30, 2025

View reviewed changes

vignettes/datatable-fread-and-fwrite.Rmd Outdated Show resolved Hide resolved

ben-schwen reviewed Jul 30, 2025

View reviewed changes

vignettes/datatable-fread-and-fwrite.Rmd Outdated Show resolved Hide resolved

ben-schwen reviewed Jul 30, 2025

View reviewed changes

vignettes/datatable-fread-and-fwrite.Rmd Outdated Show resolved Hide resolved

updated

e3f13bf

removed section 1.5

6375f71

venom1204 requested a review from ben-schwen July 31, 2025 12:39

ben-schwen reviewed Aug 4, 2025

View reviewed changes

vignettes/datatable-fread-and-fwrite.Rmd Outdated Show resolved Hide resolved

ben-schwen reviewed Aug 4, 2025

View reviewed changes

vignettes/datatable-fread-and-fwrite.Rmd Outdated Show resolved Hide resolved

ben-schwen reviewed Aug 4, 2025

View reviewed changes

vignettes/datatable-fread-and-fwrite.Rmd Show resolved Hide resolved

ben-schwen reviewed Aug 4, 2025

View reviewed changes

venom1204 and others added 4 commits August 4, 2025 15:14

Update vignettes/datatable-fread-and-fwrite.Rmd

e01472d

Co-authored-by: Benjamin Schwendinger <52290390+ben-schwen@users.noreply.github.com>

Update vignettes/datatable-fread-and-fwrite.Rmd

64b859e

Co-authored-by: Benjamin Schwendinger <52290390+ben-schwen@users.noreply.github.com>

Merge branch 'master' into issue2855

4b2dae9

added info in .rd

799d0b3

ben-schwen reviewed Aug 5, 2025

View reviewed changes

venom1204 added 2 commits August 5, 2025 09:06

updated

f210a33

Merge branch 'master' into issue2855

87b7b1c

ben-schwen reviewed Aug 5, 2025

View reviewed changes

ben-schwen requested changes Aug 5, 2025

View reviewed changes

updated cygwin part

e7d5cfe

ben-schwen approved these changes Aug 6, 2025

View reviewed changes

venom1204 requested a review from tdhock August 6, 2025 08:19

venom1204 added 2 commits August 11, 2025 08:30

added the links

9bfb5d5

Merge branch 'master' into issue2855

d1cb56a

venom1204 requested a review from ben-schwen August 11, 2025 08:31

venom1204 mentioned this pull request Aug 11, 2025

main discussion page for gsoc-2025 (VINIT) venom1204/discussion#1

Open

venom1204 requested a review from jangorecki August 11, 2025 08:35

jangorecki removed their request for review August 11, 2025 16:18

tdhock merged commit 0e17e62 into master Aug 14, 2025
11 checks passed

ChristianWia mentioned this pull request Aug 14, 2025

translate new vignette datatable-fread-and-fwrite.Rmd #7251

Closed

tdhock mentioned this pull request Aug 20, 2025

#7251 Vignette fread fwrite FR translated #7253

Merged

jangorecki deleted the issue2855 branch September 27, 2025 09:16


		#### 1.1.2 Reading from R connections and URLs

		`fread()` is highly versatile and can accept R connection objects as input to its file (or input) argument. This allows you to read from various sources, including:


		Look at this [example](https://stackoverflow.com/questions/36256706/fread-together-with-grepl/36270543#36270543) for more detail.

		On Windows we recommend [Cygwin](https://www.cygwin.com/) (run one .exe to install) which includes the command line tools such as grep. In March 2016, Microsoft [announced](https://www.hanselman.com/blog/developers-can-run-bash-shell-and-usermode-ubuntu-linux-binaries-on-windows-10) they will include these tools in Windows 10 natively. On Linux and macOS, these tools have always been included in the operating system. You can find many examples and tutorials about command line tools online. We recommend [Data Science at the Command Line](https://www.oreilly.com/library/view/data-science-at/9781491947845/).


		`fread()` doesn't directly support SQL `INSERT` scripts, but they can be processed via command-line tools. For example, given `insert_script.sql`:

		```{r, eval=FALSE}


		### 1.6 `integer64` Support

		By default, `fread` detects integers larger than 2³¹ and reads them as `bit64::integer64` to preserve full precision. This behavior can be overridden in three ways:


		When data is written as strings (either inherently, like character columns, or by choice, like `dateTimeAs="ISO"`), `quote="auto"` (default) intelligently quotes fields:

		Contextual Quoting: Only quotes fields if they contain the delimiter (`sep`), a double quote, or a newline. This ensures clean, RFC 4180-compliant CSVs.

added fwrite and fread vignette #7216

added fwrite and fread vignette #7216

Uh oh!

Conversation

venom1204 commented Jul 27, 2025

Uh oh!

codecov bot commented Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jangorecki commented Jul 27, 2025

Uh oh!

jangorecki Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aitap Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChristianWia commented Jul 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ben-schwen left a comment

Choose a reason for hiding this comment

Uh oh!

ben-schwen left a comment

Choose a reason for hiding this comment

Uh oh!

tdhock commented Aug 6, 2025

Uh oh!

tdhock commented Aug 14, 2025

codecov bot commented Jul 27, 2025 •

edited

Loading

jangorecki Jul 27, 2025 •

edited

Loading

aitap Jul 27, 2025 •

edited

Loading