Thanks for your work, it seems really good. I have pushed this to the topic/lsfd branch, we can share this branch to work on lsfd before it will be merged to the master branch (I'll work on it in the next few days too).

Notes (which list;-)):

• it would be nice to avoid generic struct names like "file", it would be better to have for example "struct lsfd_file", etc.
• can we parse "name" and deduce TYPE rather than use UNKN? , for example

         UNKN  nodev:anon_inodefs    15      11523 anon_inode:inotify   
         UNKN  nodev:anon_inodefs    15      11523 anon_inode:[signalfd]
         UNKN  nodev:anon_inodefs    15      11523 anon_inode:[timerfd]

seems like we can use INOTIFY, SIGNAL, and TIMER there

Why the same file for the same task is printed more than once:

spotify                      2959812            kzak    mem  r--  REG                sda4     0     700076 /usr/lib64/libogg.so.0.8.4
spotify                      2959812            kzak    mem  r-x  REG                sda4     0     700076 /usr/lib64/libogg.so.0.8.4
spotify                      2959812            kzak    mem  r--  REG                sda4     0     700076 /usr/lib64/libogg.so.0.8.4
spotify                      2959812            kzak    mem  ---  REG                sda4     0     700076 /usr/lib64/libogg.so.0.8.4
spotify                      2959812            kzak    mem  r--  REG                sda4     0     700076 /usr/lib64/libogg.so.0.8.4

it would be nice to add a hint about the difference to the output. Now it's confusing.

• It would be nice to support more "views", the current output is huge, and add some aggregations to reduce it would be useful in some use-cases:

  • commands/tasks -- provides complete info; (the current implementation) the relation COMMAND:INODE:etc is unique
  • file - provide a list of used inodes; INODE is unique in output, COMMAND is not printed and replaced by the number of processes (aka NPROCS - see lsns where we do the same), USER is replaced by NUSERS, etc.
  • user - provide a list of users who has open something; UID is unique in output, INODE is replaced by NFILES and COMMAND by NPROCS, etc.
  • summary - just two columns, description and value, something like:
    number of open files: 123
    number of processes: 567
    used types: REG,DIR,SOCK,INOTIFY, ....
    used block devices: sda1, sdb2, ...

• add MAJ:MIN for major and minor numbers, remove DEVICE (we use MAJ:MIN in other tools)

• add DEVTYPE for blk, char, and nodev

• remove "nodev:" from DEVNAME

• maybe rename DEVNAME to SOURCE as we have here things like 'devtmpfs' etc.

• add --deleted and --nodeleted

• add filters; yes I know, you can use grep, but for some use-cases like JSON or when you filter by data that is missing in the output it's better to have a basic filter in the command:

  • add filter by TYPE (lsfd --type REG)
  • add filter by filename/path (lsfd /usr or lsfd /etc/passwd)
  • add filter by SOURCE (lsfd --source sda1)
  • add filter by DEVTYPE (lsfd --devtype blk)
  • add filter by COMMAND (lsfd --command systemd)
  • add filter by task type, userspace vs. kerne threads -- see ps(1) and commands with [name] (lsfd --kernel | --userspace)

• filters should be possible to use with "views" (lsfd --summary --command systemd)

• man page :-)

Thank you for the comment.
I will improve the code base on the comment.

add filters; yes I know, you can use grep,

You don't know about my private plan for RHEL13 :-P

I think we should add filtering API to llibsmartcols first.
Base on the API we can add a tcp dump alike filtering language to the command line interface of ALL commands using llibsmartcols (if you want:-).
We can add lsfd --type ... or anything we want based on the language.

About lsfd, just adding the API to llibsmartcols is not enough.
Like Wireshark, there can be a "display filter" and a "capturing filter".
The API is for implementing the display filter.
The capturing filter will be part of lsfd. The capturing filter may improve the execution time of lsfd drastically.

add filters; yes I know, you can use grep,

You don't know about my private plan for RHEL13 :-P

LOL, would you please say more about? ;-)

I think we should add filtering API to libsmartcols first.

Yes, in many cases it makes sense to use the libscols_table as a container where the application keeps data (and in some cases, it's impossible, for example in lsblk the tree of data is too complex and in lscpu the data capturing is very tricky.)

Base on the API we can add a tcp dump alike filtering language to the command line interface of ALL commands using llibsmartcols (if you want:-).
We can add lsfd --type ... or anything we want based on the language.

About lsfd, just adding the API to llibsmartcols is not enough.
Like Wireshark, there can be a "display filter" and a "capturing filter".
The API is for implementing the display filter.

I more and more think about "liblsdata" to handle also data capturing part of the applications. Now many operations
are pretty simple, read from any file, convert to human-readable size, or to some predefined static string, or call snprintf(), etc. And with capture filters, we can drive it in a more elegant (effective) way without extra code in applications.

Maybe it would be also possible to replace

 switch(data_id) {
 case FOO:
      str = read_file("/soemthing");
      break
 case BAR:
      xasprinf(&str, "%d", data->bar);
      break
}

with:

lsdata {
   [FOO] = { LS_FILE, "/something", ...},
   [BAR] = { LS_STUCT, get_data_func, offsetof(struct data, bar), }
};

not sure, and maybe it's overengineering :-)

The capturing filter will be part of lsfd. The capturing filter may improve the execution time of lsfd drastically.

Yes.

Now, when I think about a way how to describe the data capturing, then it seems like a nonsense. It's better to keep all the switch(id) { case COL_FOO: } in the applications as we use now. So, ignore the previous comment :)

@masatake hi, why do you read /proc/%d/map_files/ as well as /proc/%d/maps?

It seems the "maps" file contains all the information, or is it possible that filenames in the "maps" file are truncated?

@masatake hi, why do you read /proc/%d/map_files/ as well as /proc/%d/maps?

Just my mistake. The code accessing files under map_files should be removed.

@karelzak, could you look at https://github.com/masatake/fd-catalog?
Before improving lsfd, I think we need a good test situation(?) generator.

@karelzak, could you look at https://github.com/masatake/fd-catalog?
Before improving lsfd, I think we need a good test situation(?) generator.

That's nice! It would be nice to have it in tests/helpers/ directory :-)

In the next few days, I will rewrite lib/procutils.c to make it more usable in lsfd (and other tools).

I'd like to have something like we already have for sysfs (lib/sysfs.c); it means code based on process directory file descriptor and "*at" functions to reuse lib/path.c functions.

The more than half of items you listed as review comments about the desing of columns.
I would like to write how I thought when developing lsfd.

Based on studying lsof, I planed to design the columns in two levels.
The first level is for providing raw data for machine processing.
One of the critical advantages of being part of util-linux is that libsmarcols provides json output.
The feature allows peope to develop post processors.
For the purpose, I think lsfd should be able to emit raw data.
e.g. I used "UNKN" value for "TYPE" column. I introduced MODE column. In lsof, the mode values are unified to file descriptors like "1r", "2w", "3u".

The second leve is for providing wel-coocked data for users.
The raw data is too complicated and redandunt for our eyes.
More readable coulmns should be provided and we should enable them as default column sets.

I could not complete lsfd before opening this pull request. So the levels are not separated well in
my pull request. However, I had a time I will use as _ prefix for the names of level 1, raw data columns, and hide them.

My idea of "levels" are very related to your "view".

My understanding is you are working on merging lsfd with modifying it acceptable.
I cannot say I have much time but till merging becomes ready, I will work on the fd-catalog and filtering feature based on libsmartcol.

Oh, I misunderstood what were happning. You are already working of the lsfd branch. I will watch it.

For now, I'm working on an elegant way how to read data from /proc/pid/ to reduce effort and code duplication in util-linux. The current code in lib/procutils.c is poor and I'd like to have something better (like we already have for sysfs).

So, my current intention is not to change how lsfd prints data on output. It will be the next step and I wait for you with it ;-)

So, my current intention is not to change how lsfd prints data on output. It will be the next step and I wait for you with it ;-)

Do you mean I should start making more commits based on your reivew comments?

So, my current intention is not to change how lsfd prints data on output. It will be the next step and I wait for you with it ;-)

Do you mean I should start making more commits based on your reivew comments?

Yes, but use topic/lsfd branch if you can.

Yes, but use topic/lsfd branch if you can.

I see.

I got one more question about the way to work.
How should I provide new changes to the branch?
Should I make a new pull request?

Maybe you can try "git push --force" to this PR. (I mean git push --force to your masatake:lsfd on github.)

I see. Thank you. I will keep this full request fresh.

I have committed other changes to collect_* stuff. The last remaining stuff is maps and mountinfo. I'll do it tomorrow. We need to parse mountinfo only once for the mount namespace. It's overkill to do it for each process.

We should reflect the result of filtering to the exit status: lsof-org/lsof#128.

The topic/lsfd branch updated, thanks!

Sorry for the delay last week (I had a vacation and after then I had an issue with my workstation disk...).

Today I committed improved /proc/#/maps code, now it does not use /proc/#/map_files/ symlinks.

For this week my TODO is to reduce the number of situations we call stat(), because there are so many duplications

$ ./lsfd -oNAME | wc -l
37982
$ ./lsfd -oNAME | sort -u | wc -l
1761

It would be probably better to have two structs

1. lsfd_file - file itself; with path, struct stat, etc.
2. lsfd_fileref - a reference to the file; with process-specific data like mmap stuff, fd, etc.

The initial version of the display filter engine works.
Based on it I can implement the filter options you suggested.

There was a choice implementing filter options based on a filtering engine working in the stage of information-collecting time (collecting filter). For making the execution time of lsfd shorter, the collecting filter is much better than the display filter. However, the collecting filter may be tightly coupled with the core of lsfd. It is much hard to implement the collecting filter than the display filter.

On the other hand, we can implement the display filter based on libsmarcols. We don't need any knowledge about lsfd to implement the display filter.

Here is an example session:

#  ./lsfd -D '(PID == 1) && (ASSOC == "exe")'
COMMAND PID USER ASSOC MODE TYPE SOURCE MNTID   INODE NAME
systemd   1 root   exe  ---  REG   dm-0     0 1847406 /usr/lib/systemd/systemd
#  ./lsfd -D '(PID == 1) && ((ASSOC == "exe") or (ASSOC == "1"))'
COMMAND PID USER ASSOC MODE TYPE SOURCE MNTID   INODE NAME
systemd   1 root   exe  ---  REG   dm-0     0 1847406 /usr/lib/systemd/systemd
systemd   1 root     1  rw-  CHR  mem:3    25       4 /dev/null
#  ./lsfd -D '(PID == 1) && ((ASSOC == "exe") or (ASSOC == "1") or (SOURCE == "tmpfs"))'
COMMAND PID USER ASSOC MODE TYPE SOURCE MNTID   INODE NAME
systemd   1 root   exe  ---  REG   dm-0     0 1847406 /usr/lib/systemd/systemd
systemd   1 root     1  rw-  CHR  mem:3    25       4 /dev/null
systemd   1 root   120  rw- FIFO  tmpfs    29     752 /run/dmeventd-server
systemd   1 root   121  rw- FIFO  tmpfs    29     753 /run/dmeventd-client
systemd   1 root   399  rw- FIFO  tmpfs    29     757 /run/initctl
# ./lsfd -D '(PID == 1) && ((ASSOC == "exe") or (ASSOC == "1") or (TYPE == "FIFO"))'
COMMAND PID USER ASSOC MODE TYPE SOURCE MNTID   INODE NAME
systemd   1 root   exe  ---  REG   dm-0     0 1847406 /usr/lib/systemd/systemd
systemd   1 root     1  rw-  CHR  mem:3    25       4 /dev/null
systemd   1 root   120  rw- FIFO  tmpfs    29     752 /run/dmeventd-server
systemd   1 root   121  rw- FIFO  tmpfs    29     753 /run/dmeventd-client
systemd   1 root   391  r-- FIFO pipefs    14   18206 pipe:[18206]
systemd   1 root   399  rw- FIFO  tmpfs    29     757 /run/initctl
./lsfd -J -D '(PID == 1) && (ASSOC == "exe")'
# ./lsfd -J -D '(PID == 1) && (ASSOC == "exe")'
{
   "lsfd": [
      {
         "command": "systemd",
         "pid": 1,
         "user": "root",
         "assoc": "exe",
         "mode": "---",
         "type": "REG",
         "source": "dm-0",
         "mntid": 0,
         "inode": 1847406,
         "name": "/usr/lib/systemd/systemd"
      }
   ]
}
# ./lsfd -D 'NAME == "/etc/passwd"'
COMMAND        PID   USER ASSOC MODE TYPE SOURCE MNTID  INODE NAME
fd-catalog 1677961 yamato     3  r--  REG   dm-0    88 271210 /etc/passwd

Unlike using grep, the table is enough compact to the data.

I will polish the display filter code.

There was a choice implementing filter options based on a filtering engine working in the stage of information-collecting time (collecting filter). For making the execution time of lsfd shorter, the collecting filter is much better than the display filter. However, the collecting filter may be tightly coupled with the core of lsfd. It is much hard to implement the collecting filter than the display filter.

I think it's better to start with a display filter. I don't think we need a collecting filter which able to evaluate individual entries. It seems better to have a way how to enable/disable some parts of the data collecting. For example, disable memory maps parsing, disable /proc/#/fds, or disable socket, etc. ... then we can later evaluate display filter and optimize collecting.

For example for display filter ASSOC == "exe" or ASSOC == "1" you can disable /proc/#/maps parsing at all.

On the other hand, we can implement the display filter based on libsmartcols. We don't need any knowledge about lsfd to implement the display filter.

This is an interesting idea, because it is something that we can use in all libsmartcols tools ;-)

Unlike using grep, the table is enough compact to the data.

I guess people will argue against display filters by awk where you can compare columns, but the problem with grep and awk is that it uses displayed data, it means already formatted, etc. The JSON is a good example. Another thing is that applications can also use display filter to optimize data collecting (my previous note).

Yes, devno for btrfs sucks for years ... (see libmount/findmnt where we have extra optimizations for btrfs); there is an attempt to fix it in lkml, but not sure what is the current status.

@karelzak, I pushed the changes adding libsmartcols based filtering. I hope you like them.

Like lsfd itself, the changes may require your massive revising
I think the filtering code should be part of libsmartcols, not lsfd.
I tried to make the code independent from lsfd in the most of parts.
However, I have had one lsfd specific assumption: a negative number is not needed in the filter expression.

At the first glance, the output filter seems cool -- I do not test it yet, that's my plan for tomorrow and I'll probably merge it to topic/lsfd after that, ok?

I've tried to split "struct file" into "struct file" and "struct file_reference" to share information about files between processes (to reduce stat() calls), but the result was unsuccessful. The Linux stat() is so fast for already open files that the benefit of stat() calls reduction is the same as overhead introduced by a maintenance of the global files list in lsfd :-)

But I have found a more effective way how to reduce 50% of stat() calls, in /proc/#/maps the same file is usually mapped more than once, so reuse the previous "struct file" is simple without any global list of files -- see commit cf2d4b1.

To implement --summary, I'm working on adding counter feature to libsmartcol infra.
I report that the initial prototype worked.

# : printing build-in default counters
#  ./lsfd -S
VALUE COUNTER
  371 processes
  207 root owned processes
  138 kernel threads
 4579 open files
 1210 RO open files
  287 WO open files
 2272 shared mappings
 2203 RO shared mappings
    0 WO shared mappings
 1086 regular files
    7 directories
 1591 sockets
  419 fifos/pipes
  621 character devices
    0 block devices
  855 unknonw types
# : default counters with a filter
#  ./lsfd -Q '(PID == 1)' -S
VALUE COUNTER
    1 processes
    1 root owned processes
    0 kernel threads
  229 open files
   10 RO open files
    1 WO open files
    2 shared mappings
    2 RO shared mappings
    0 WO shared mappings
    2 regular files
    1 directories
  174 sockets
    4 fifos/pipes
   18 character devices
    0 block devices
   30 unknonw types
# : custom counters with a filter 
# ./lsfd -Q '(PID == 1)' --counter="systemd's sockets":'(TYPE == "SOCK")' --counter="systemd's char devs":'(TYPE == "CHR")' -S
VALUE COUNTER
  174 systemd's sockets
   18 systemd's char devs
# ./lsfd --counter="MAPLEN >= 1":'(MAPLEN >= 1)' --counter="MAPLEN >= 10":'(MAPLEN >= 10)' --counter="MAPLEN >= 100":'(MAPLEN >= 100)' --counter="MAPLEN >= 1000":'(MAPLEN >= 1000)' --counter="MAPLEN >= 10000":'(MAPLEN >= 10000)' --counter="MAPLEN >= 100000":'(MAPLEN >= 100000)' -S
VALUE COUNTER
58517 MAPLEN >= 1
19011 MAPLEN >= 10
 5885 MAPLEN >= 100
 2064 MAPLEN >= 1000
  193 MAPLEN >= 10000
    0 MAPLEN >= 100000
# : json output
# ./lsfd -J -Q '(PID == 1)' -S | head
{
   "lsfd-summary": [
      {
         "value": 1,
         "counter": "processes"
      },{
         "value": 1,
         "counter": "root owned processes"
      },{
         "value": 0,    

The current implementation archives:

• summary - just two columns, description and value, something like:
  • number of open files: 123
  • number of processes: 567

However, it doesn't:

• summary - just two columns, description and value, something like:
  • used types: REG,DIR,SOCK,INOTIFY, ....
  • used block devices: sda1, sdb2, ...

To implement these, we need the "pivot" table feature as we see in spreadsheet software like Excel.

@karelzak, up to a6a8fd0, the changes are ready to be reviewed on my side.
Could you merge them if they are acceptable?

Merged, thanks.

• Would be possible to improve filter parsing to not require '( ... )' between expressions?

./lsfd -Q 'USER=="kzak" && SOURCE=="sda3"'
lsfd: error: unexpected right operand type STR for: &&

but

./lsfd -Q '(USER=="kzak") && (SOURCE=="sda3")'

works as expected.

• ad --counter, if I good understand, the idea is to implement it as an aggregator that is possible to restrict by an expression, so synopsis will be:

  --counter=<label>:<expression>

and the default for -S, --summary will be some predefined counters with human-readable strings. Sounds good.

I don't think we need pivot table, just compose an array of strings (see include/strv.h) and before printing convert it to
a comma (or line-break) separated list of strings. But my suggestion is to postpone this problem and merge the counter without this feature. We can implement it later.

I found implementing the feature, what I called "pivot," is not so easy.
So, this time, I submitted the simple counter implementation.

Would be possible to improve filter parsing to not require '( ... )' between expressions?

It is not easy for me to improve it.

Merged.

I have also done small changes to the --summary semantic to make it more user-friendly (make --summary=only the default when argument not specified). I'll also improve JSON support in libsmartcols to support more tables in one JSON object.

@karelzak, thank you for reviewing and merging.

There are still many TODO items listed in the first comment in this pull request, but lsfd can be used for daily work already.
This pull request becomes too long. So I would like to close this.

Hereafter, I would like to open separate pull requests topic by topic.

Though I am a member of this organization, I would like to use a topic branch of my repository (masatake/util-linux) to make a pull request. I'm not sure this method is popular or not. I have used this method in the project I have maintained (universal-ctags/ctags), and it is comfortable for me.

For a while, when making a pull request, I will target "topic/lsfd" branch of "util-linux" repository in "util-linux" organization for merging. If you have an idea about the target branch, let me know.

Though this pull request will be closed, I will maintain the TODO lists in the first comment.

If you find a critical issue for including lsfd in the next release, let me know.

I think we can merge topic/lsfd to the master branch and you can continue with your changes and PRs against the master branch. It will help us to gather more feedback from others developers and users.

The code is good enough, release early, release often ... ;-)

My plan is to mark lsfd as "experimental" for the next release. It means we can change command-line options and semantic of the command without caring about backward compatibility.

I think we can merge topic/lsfd to the master branch and you can continue with your changes and PRs against the master branch.

I see. I will do so.

Merged to the master branch.