{"id":1171,"date":"2023-07-20T18:32:00","date_gmt":"2023-07-20T13:02:00","guid":{"rendered":"https:\/\/geekpython.in\/?p=1171"},"modified":"2024-03-01T17:11:36","modified_gmt":"2024-03-01T11:41:36","slug":"fileinput-module","status":"publish","type":"post","link":"https:\/\/geekpython.in\/fileinput-module","title":{"rendered":"How to Read Multiple Files Simultaneously With fileinput Module In Python"},"content":{"rendered":"\n

The fileinput<\/code> module is a part of the standard library and is used when someone needs to iterate the contents of multiple files simultaneously. Well, Python’s in-built open()<\/code> function can also be used for iterating the content but for only one file at a time.<\/p>\n\n\n\n

You’ll explore the classes and functions provided by the fileinput<\/code> module to iterate over multiple files.<\/p>\n\n\n\n

But one thing, you could use fileinput<\/code> to iterate the single file also, but it would be better to use the open()<\/code> function for it.<\/p>\n\n\n\n

<\/a>Basic Usage<\/h2>\n\n\n\n
import fileinput\n\n# Creating fileinput instance and passing multiple files\nstream = fileinput.input(files=('test.txt',\n                                'sample.txt',\n                                'index.html'))\n\n# Iterating the content\nfor data in stream:\n    print(data)<\/pre><\/div>\n\n\n\n
The fileinput<\/code> module was first imported, and then the fileinput<\/code> instance was created by calling fileinput.input()<\/code> and passing the tuple of files (test.txt<\/code>, sample.txt<\/code>, and index.html<\/code>). This will result in the return of an iterator.<\/p>\n\n\n\n
The contents of the files were then iterated and printed using the for<\/code> loop.<\/p>\n\n\n\n
Hi, I am a test file.\nHi, I am a sample file for testing.\n<html lang=\"en\">\n<head>\n    <title>Test HTML File<\/title>\n<\/head>\n<body>\n<h1>Hi, I am a simple HTML File.<\/h1>\n<\/body>\n<\/html><\/pre><\/div>\n\n\n\n
Another approach would be to use the fileinput<\/code> module as a context manager<\/a>. This method is somewhat safe because it ensures that the fileinput<\/code> instance is closed even if an exception occurs.<\/p>\n\n\n\n
import fileinput\n\nwith fileinput.input(files=('test.txt', 'sample.txt')) as files:\n    for data in files:\n        print(data)<\/pre><\/div>\n\n\n\n
In the above demonstration, the fileinput<\/code> module was used as a context manager with the 'with'<\/code> statement.<\/p>\n\n\n\n
The above code will return an iterator and will assign it to the files<\/code> variable (due to the as<\/code> clause) then the data will be iterated using the files<\/code> variable.<\/p>\n\n\n\n
Hi, I am a test file.\nHi, I am a sample file for testing.<\/pre><\/div>\n\n\n\n
<\/a>The fileinput.input() Function<\/h2>\n\n\n\n
The fileinput.input()<\/code> function is the primary interface of the fileinput<\/code> module, by using it, the purpose of using the fileinput<\/code> module is nearly fulfilled. You saw a glimpse of the fileinput.input()<\/code> function in the previous section, this time, you’ll learn more about it.<\/p>\n\n\n\n
<\/a>Syntax<\/h3>\n\n\n\n
fileinput.input(files=None, inplace=False, backup='', mode='r', openhook=None, encoding=None, errors=None)<\/code><\/p>\n\n\n\n
Parameters:<\/strong><\/p>\n\n\n\n
files<\/code>: Defaults to None<\/code>. Takes a single file or multiple files to be processed.<\/p>\n\n\n\n
inplace<\/code>: Defaults to False<\/code>. When set to True<\/code>, the files can be modified directly.<\/p>\n\n\n\n
backup<\/code>: Defaults to an empty string. The extension is specified for the backup files when inplace<\/code> is set to True<\/code>.<\/p>\n\n\n\n
mode<\/code>: Default to read mode. This can only open files in read mode hence, we can open the file in r<\/code>, rb<\/code>, rU<\/code>, and U<\/code>.<\/p>\n\n\n\n
openhook<\/code>: Defaults to None<\/code>. A custom function for controlling how files are opened.<\/p>\n\n\n\n
encoding<\/code>: Defaults to None<\/code>. Specifies the encoding to be used to read the files.<\/p>\n\n\n\n
errors<\/code>: Defaults to None<\/code>. Specifies how the errors should be handled.<\/p>\n\n\n\n
<\/a>Modifying the Files Before Reading<\/h3>\n\n\n\n
import fileinput\n\nwith fileinput.input(files=('test.txt', 'sample.txt'), inplace=True) as files:\n    for data in files:\n        modified_content = data.lower()\n        print(modified_content)<\/pre><\/div>\n\n\n\n
The parameter inplace<\/code> is set to True<\/code> in the above code, which enables the editing of the file before reading.<\/p>\n\n\n\n
The upper code will lowercase the content present inside both files (test.txt<\/code> and sample.txt<\/code>).<\/p>\n\n\n\n
<\/a>Storing Backup of Files<\/h3>\n\n\n\n
When the inplace<\/code> parameter is set to True<\/code>, the original files can be edited, but the original state of the files can be saved in another file using the backup<\/code> parameter.<\/p>\n\n\n\n
import fileinput\n\nwith fileinput.input(files=('test.txt', 'sample.txt'),\n                     inplace=True, backup='.bak') as files:\n    for data in files:\n        modified_content = data.capitalize()\n        print(modified_content)<\/pre><\/div>\n\n\n\n
The above code will capitalize the content and the original files will be saved as test.txt.bak<\/code> and sample.txt.bak<\/code> due to the backup='.bak'<\/code>.<\/p>\n\n\n\n
<\/a>Controlling the Opening of the File<\/h3>\n\n\n\n
import fileinput\n\ndef custom_open(filename, mode):\n    data = open(filename, \"a+\")\n    data.write(\" Data added through function.\")\n    return open(filename, mode)\n\nwith fileinput.input(files=(\"test.txt\", \"sample.txt\"), openhook=custom_open) as file:\n    for data in file:\n        print(data)<\/pre><\/div>\n\n\n\n
The custom_open()<\/code> function is defined that takes two parameters filename<\/code> and mode<\/code>. The function opens the file in append + read<\/code> mode and then writes the string and returns the file object.<\/p>\n\n\n\n
\n
The hook must be a function that takes two arguments, filename<\/code> and mode<\/code>, and returns an accordingly opened file-like object.Source<\/a><\/p>\n<\/blockquote>\n\n\n\n
The files are then passed to the fileinput.input()<\/code> function, and the openhook<\/code> parameter is set to custom_open<\/code>. The custom_open()<\/code> function will be in charge of opening the files. The file content was iterated and printed.<\/p>\n\n\n\n
Hi, i am a test file. Data added through function.\nHi, i am a sample file for testing. Data added through function.<\/pre><\/div>\n\n\n\n
<\/a>Reading Unicode Characters<\/h3>\n\n\n\n
You have a file having Unicode characters and need to read that file, to read Unicode characters, specific encodings are used.<\/p>\n\n\n\n
with fileinput.input(files=('test_unicode.txt'), encoding='utf-8') as files:\n    for data in files:\n        print(data)<\/pre><\/div>\n\n\n\n
The UTF-8 encoding can be used to read the Unicode characters, hence, the encoding<\/code> parameter is set to utf-8<\/code> encoding.<\/p>\n\n\n\n
\ud83d\ude01\ud83d\ude02\ud83d\ude05<\/pre><\/div>\n\n\n\n
<\/a>Handling Errors<\/h3>\n\n\n\n
To handle the error, use the errors<\/code> parameter. Take the above code as an example: if the encoding<\/code> was not specified, the code would throw a UnicodeError<\/code>.<\/p>\n\n\n\n
with fileinput.input(files=('test_bin.txt'), errors='ignore') as files:\n    for data in files:\n        print(data)\n\n----------\n\u00f0\u0178\u02dc\u00f0\u0178\u02dc\u201a\u00f0\u0178\u02dc\u2026<\/pre><\/div>\n\n\n\n
The errors<\/code> parameter is set to ignore<\/code>, which means that the error will be ignored. The errors<\/code> parameter can also be set to strict<\/code> (raise an exception if an error occurs) or replace<\/code> (replace an error with a specified error).<\/p>\n\n\n\n
<\/a>Functions to Access Input File Information<\/h3>\n\n\n\n
There are some functions that can be used to access the information of the input files which are being processed using the fileinput.input()<\/code> function.<\/p>\n\n\n\n
Getting the File Names<\/strong><\/p>\n\n\n\n
Using the fileinput.filename()<\/code> function, the name of the currently processed files can be displayed.<\/p>\n\n\n\n
with fileinput.input(files=('test.txt', 'sample.txt')) as files:\n    for data in files:\n        print(f\"File: {fileinput.filename()}\")\n        print(data)<\/pre><\/div>\n\n\n\n
Output<\/strong><\/p>\n\n\n\n
File: test.txt\nHi, i am a test file. Data added through function. Added data to the file.\nFile: sample.txt\nHi, i am a sample file for testing. Data added through function. Added data to the file.<\/pre><\/div>\n\n\n\n
Getting the File Descriptor and Line and File Line Number<\/strong><\/p>\n\n\n\n
The fileinput.fileno()<\/code> function returns the active file’s file descriptor, the fileinput.lineno()<\/code> function returns the cumulative line number, and the fileinput.filelineno()<\/code> function returns the line number of the currently processed file.<\/p>\n\n\n\n
with fileinput.input(files=('test.txt', 'sample.txt')) as files:\n    for data in files:\n        print(f\"{fileinput.filename()}'s File Descriptor: {fileinput.fileno()}\")\n        print(f\"{fileinput.filename()}'s File Line Number: {fileinput.filelineno()}\")\n        print(f\"{fileinput.filename()}'s File Cumulative Line No.: {fileinput.lineno()}\")<\/pre><\/div>\n\n\n\n
Output<\/strong><\/p>\n\n\n\n
test.txt's File Descriptor: 3\ntest.txt's File Line Number: 1\ntest.txt's File Cumulative Line No.: 1\n\nsample.txt's File Descriptor: 3\nsample.txt's File Line Number: 1\nsample.txt's File Cumulative Line No.: 2<\/pre><\/div>\n\n\n\n
Checking Reading Status<\/strong><\/p>\n\n\n\n
with fileinput.input(files=('test.txt', 'sample.txt')) as files:\n    for data in files:\n        print(f\"Read First Line: {fileinput.isfirstline()}\")\n        print(f\"Last Line Read From sys.stdin: {fileinput.isstdin()}\")\n\n----------\nRead First Line: True\nLast Line Read From sys.stdin: False\nRead First Line: True\nLast Line Read From sys.stdin: False<\/pre><\/div>\n\n\n\n
The fileinput.isfirstline()<\/code> function returns True<\/code> if the line read from the current file is the first line otherwise returns False<\/code>, since both files contain a single line, it returned True<\/code>.<\/p>\n\n\n\n
When the last line of the input file was read from sys.stdin<\/code>, the fileinput.isstdin()<\/code> function returns True<\/code>, otherwise, it returns False<\/code>.<\/p>\n\n\n\n
Closing the File<\/strong><\/p>\n\n\n\n
When using fileinput.input()<\/code> function as the context manager with the with<\/code> statement, the file closes anyway but fileinput.close()<\/code> function is also used to close the resources when the work is done.<\/p>\n\n\n\n
import fileinput\n\nwith fileinput.input(files=('test.txt', 'sample.txt')) as file:\n    for data in file:\n        if data > data[:26]:\n            fileinput.close()\n            print('File has more than 25 characters.')\n        else:\n            print(data)<\/pre><\/div>\n\n\n\n
The above code demonstrates the use of the fileinput.close()<\/code> function, which closes the file if it contains more than 25 characters and prints a message otherwise the content is printed.<\/p>\n\n\n\n
File has more than 25 characters.<\/pre><\/div>\n\n\n\n
However, because the file contained more than 25 characters, the file was closed and the message was printed.<\/p>\n\n\n\n
<\/a>The FileInput Class<\/strong><\/h2>\n\n\n\n
The fileinput.FileInput<\/code> class is an object-oriented alternative to the fileinput.input()<\/code> function. The parameters are identical to those of the input()<\/code> function.<\/p>\n\n\n\n
Syntax<\/strong><\/p>\n\n\n\n
fileinput.FileInput(files=None, inplace=False, backup='', mode='r', openhook=None, encoding=None, errors=None)<\/code><\/p>\n\n\n\n
Example<\/strong><\/p>\n\n\n\n
import fileinput\n\nclass OpenMultipleFiles:\n    def __init__(self, *args):\n        self.args = args\n\n    def custom_open(self, filename, mode):\n        data = open(filename, \"a+\")\n        data.write(\" Added data to the file.\")\n        return open(filename, mode)\n\n    def read(self):\n        with fileinput.FileInput(files=(self.args), openhook=OpenMultipleFiles().custom_open) as file:\n            for data in file:\n                print(data)\n\nobj = OpenMultipleFiles('test.txt', 'sample.txt')\nobj.read()<\/pre><\/div>\n\n\n\n
The class OpenMultipleFiles<\/code> is defined in the above code. The class has an __init__<\/code> method that takes variadic arguments.<\/p>\n\n\n\n
A custom_open<\/code> method is defined within the class that opens the file in append+read<\/code> mode, writes some data to the file, and returns the file object.<\/p>\n\n\n\n
The read<\/code> method is defined and within the read<\/code> method the instance of the fileinput.FileInput<\/code> is created and passed the self.args<\/code> as the files argument and the openhook<\/code> parameter is set to OpenMultipleFiles().custom_open<\/code>. The contents of the files are then iterated and printed.<\/p>\n\n\n\n
Finally, the OpenMultipleFiles<\/code> class instance is created and passed the file names (test.txt<\/code> and sample.txt<\/code>) and stored within the obj<\/code> variable. The read<\/code> method is then invoked on the obj<\/code> to read the specified files.<\/p>\n\n\n\n
Hi, i am a test file. Data added through function. Added data to the file.\nHi, i am a sample file for testing. Data added through function. Added data to the file.<\/pre><\/div>\n\n\n\n
<\/a>Comparison<\/h2>\n\n\n\n
Let’s see how long it takes to process the contents of multiple files at the same time using the open()<\/code> and the fileinput.input()<\/code> function.<\/p>\n\n\n\n
import timeit\n\n# open() Function Code\ncode = '''\nwith open('test.txt') as f1, open('sample.txt') as f2:\n    f1.read()\n    f2.read()\n'''\n\nprint(f\"Open Function Benchmark: {timeit.timeit(stmt=code, number=1000)}\")\n\n# fileinput Code\nsetup = 'import fileinput'\n\ncode = '''\nwith fileinput.input(files=('test.txt', 'sample.txt')) as file:\n    for data in file:\n        data\n'''\nprint(f\"Fileinput Benchmark: {timeit.timeit(setup=setup, stmt=code, number=1000)}\")<\/pre><\/div>\n\n\n\n
Using the timeit<\/code> module, the above code measures the time it takes to process the contents of multiple files 1000 times for the fileinput.input()<\/code> function and open()<\/code> function. This method will aid in determining which is more efficient.<\/p>\n\n\n\n
Open Function Benchmark: 0.3948998999840114\nFileinput Benchmark: 0.4962893000047188<\/pre><\/div>\n\n\n\n
<\/a>Limitations<\/h2>\n\n\n\n
Every module is powerful in its own right, but it also has limitations, such as the fileinput<\/code> module.<\/p>\n\n\n\n