CLI to recursively search and transform text files in xargs style
- search and substitute (replace) text in file contents or file paths in place
- installable without sudo (using pip)
This tool was developed as an opinionated (partial) alternative to the following CLI tools.
findto recursively find file pathsfdto recursively find file paths respecting .gitignore (fdis internally used byxunderandxbatchin default if installed)grepto search textrgto search textagto search textsedto modify textxargsto repeat similar to for-looptreeto recursively check file pathsduto check disk usagecsvkitto search (and modify) text in csv filesspyqlto search (and modify) text in csv filesclickhouse-localpolars-clito search text in csv filescsvlensto view csv filesrich-clito view csv files
This tool is quicker to write although execution might be slower depending on the amount of your text data.
Prerequisite: Python 3 (3.8 or later recommended)
pip install recurtx
This is recommended only if you want to modify the source code.
git clone https://github.com/Minyus/recurtx.git
cd recurtx
python setup.py developRun any scripts for each file under a directory recursively.
Run wc -l {FILEPATH} for each file under directory_foo recursively:
xunder directory_foo "wc -l"
Quoting for the script can be omitted for most cases.
xunder directory_foo wc -l
Caveat: int, float, tuple, list, dict could be formatted unexpectedly (by fire package), for example:
00(recognized as int by Python) will be converted to0while"00"(recognized as str by Python) will be kept as is
NAME
xunder
SYNOPSIS
xunder PATH <flags> [SCRIPTS]...
POSITIONAL ARGUMENTS
PATH
Type: str
SCRIPTS
Type: str
FLAGS
--glob=GLOB
Type: str
Default: '**/*'
--replace_str=REPLACE_STR
Type: str
Default: '@@'
--show_paths=SHOW_PATHS
Type: bool
Default: False
--show_scripts=SHOW_SCRIPTS
Type: bool
Default: False
NOTES
You can also use flags syntax for POSITIONAL ARGUMENTS
Run any scripts for a batch of files in a directory recursively.
Concatenate all the contents in directory_foo.
xbatch directory_foo cat
NAME
xbatch
SYNOPSIS
xbatch PATH <flags> [SCRIPTS]...
POSITIONAL ARGUMENTS
PATH
Type: str
SCRIPTS
Type: str
FLAGS
--glob=GLOB
Type: str
Default: '**/*'
--replace_str=REPLACE_STR
Type: str
Default: '@@'
--show_paths=SHOW_PATHS
Type: bool
Default: False
--show_scripts=SHOW_SCRIPTS
Type: bool
Default: False
NOTES
You can also use flags syntax for POSITIONAL ARGUMENTS
Search a keyword, which may include wildcards, in the text file content, and optionally substitute (replace).
Search keyword_bar in each file under directory_foo recursively:
xunder directory_foo xsearch keyword_bar
Search keyword_bar and substitute (replace) with keyword_baz in each file under directory_foo recursively:
xunder directory_foo xsearch keyword_bar --sub keyword_baz
NOTES
You can also use flags syntax for POSITIONAL ARGUMENTS
NAME
xsearch - Search a keyword, which may include wildcards, in the text file content, and optionally substitute (replace).
SYNOPSIS
xsearch TARGET PATH <flags>
DESCRIPTION
Search a keyword, which may include wildcards, in the text file content, and optionally substitute (replace).
POSITIONAL ARGUMENTS
TARGET
Type: str
PATH
Type: str
FLAGS
--sub=SUB
Type: Optional[typing.Unio...
Default: None
-w, --wildcard=WILDCARD
Type: str
Default: '*'
--separator=SEPARATOR
Type: str
Default: '/'
-v, --verbose=VERBOSE
Type: int
Default: 1
-c, --context=CONTEXT
Type: typing.Union[int, NoneType]
Default: 1
-p, --plain=PLAIN
Type: bool
Default: False
NOTES
You can also use flags syntax for POSITIONAL ARGUMENTS
Find a keyword, which may include wildcards, in the file path, and optionally substitute (replace).
Search keyword_bar in each file path under directory_foo recursively:
xunder directory_foo xfind keyword_bar
Search keyword_bar and substitute (replace) with keyword_baz in each file path under directory_foo recursively:
xunder directory_foo xfind keyword_bar --sub keyword_baz
NAME
xfind
SYNOPSIS
xfind TARGET PATH <flags>
POSITIONAL ARGUMENTS
TARGET
Type: str
PATH
Type: str
FLAGS
-s, --sub=SUB
Type: Optional[str]
Default: None
-w, --wildcard=WILDCARD
Type: str
Default: '*'
-v, --verbose=VERBOSE
Type: int
Default: 1
NOTES
You can also use flags syntax for POSITIONAL ARGUMENTS
Alternative to ls -lah and du.
Show approximate total size and max size in Bytes to respond quickly in default while it takes time to run du in a big directory including many files.
Show number of files and the most common file extention(s) as well.
Run under directory_foo non-recursively (up to depth 1):
xll directory_foo
Run under directory_foo recursively up to depth 2:
xll -d 2 directory_foo
NAME
xll - Compute statistics for the directory recursively.
SYNOPSIS
xll <flags> [PATHS]...
DESCRIPTION
Compute statistics for the directory recursively.
POSITIONAL ARGUMENTS
PATHS
Type: str
FLAGS
-d, --depth=DEPTH
Type: int
Default: 1
-u, --unit_glob=UNIT_GLOB
Type: str
Default: '**/*'
-t, --type=TYPE
Type: Optional[typing.Unio...
Default: None
-g, --glob=GLOB
Type: str
Default: '**/*'
-r, --regex=REGEX
Type: str
Default: '^(?!.*(\\.git\\/|__pycache__\\/|\\.ipynb_checkpoints\\/|\\....
-n, --number_limit=NUMBER_LIMIT
Type: int
Default: 100
-s, --sort_paths=SORT_PATHS
Type: str
Default: 'asc'
-i, --info=INFO
Type: bool
Default: True
-e, --extension_most_common=EXTENSION_MOST_COMMON
Type: int
Default: 1
Read and transform tabular data using pandas.
Regarding options, see the documents for pandas.read_xxx such as:
Data types supported by pandas (not all were tested):
- "pickle"
- "table"
- "csv"
- "fwf"
- "clipboard"
- "excel"
- "json"
- "html"
- "xml"
- "hdf"
- "feather"
- "parquet"
- "orc"
- "sas"
- "spss"
- "sql_table"
- "sql_query"
- "sql"
- "gbq"
- "stata"
pip install pandas
Read files supported by pandas (such as csv and json) under directory_foo and concatenate:
xbatch directory_foo xpandas
Read and transform tabular data using polars.
Regarding options, see the documents for polars.scan_xxx (or polars.read_xxx if scan function is not available), such as:
Data types supported by polars (not all were tested):
- "csv"
- "ipc"
- "parquet"
- "database"
- "json"
- "ndjson"
- "avro"
- "excel"
- "delta"
pip install polars
Read files supported by polars (such as csv and json) under directory_foo and concatenate:
xbatch directory_foo xpolars