add age & machine and readable outputs for ldms_ls: json/tab #1568
Conversation
|
@baallan, I'm assuming that this is to make the ldms_ls output easier to parse? Let's bring up this issue with the UG and see what they think. I'm fairly certain that there are many groups parsing this output. So for example: CSV, quoting some column values, putting a comment character in front of the heading and footer lines, may also be good options. So I think that this is a really good idea, but let's get some input on the various formatting options. I would think that JSON is an obviously good option, so maybe we do that first and then add the other formats in separate pull requests? |
|
@tom95858 the json and tab formats are both ready to go, specified with -f tab or -f json. All the permutations and extensions you mention are easily accommodated either by adding additional values other than 'tab' and 'json' (in another pr) or immediately by running the tab output through sed/awk/grep to replace tabs/headers as desired. That said, no harm discussing at a UG meeting. I'm appending some examples of why tab delimited output is immediately interesting to folks writing production scripts that manage ldmsd. So immediate, that it may want back- interesting examples using the current implementation: get the schema name listget the schemas sorted by length of time to get the sampleget the sets late in reporting (often due to an agg issue)get the interval and offset last hinted; check that agg config is working as expected |
|
@baallan, the only one that gives me pause is TAB? Why not CSV instead? |
Unix utilities (cut, sort, etc) work on tab separators by default. But that said, there is an ietf4180 standard that was added to the csv store and could easily be added to the ldms_ls -f option list, also. I would prefer not to hold up getting json/tab outputs for it, though. |
|
An example (with some metrics elided for length) of the json format with the -l option and after that with both -l -v: with -l -v (again metrics elided for length) |
|
And an example of the tab output without filters (-vvl -f tab). |
|
@baallan I found that the The output for meminfo is correct. |
| .nf | ||
| .RS | ||
| ldms_ls -h vm1 -x sock -p 60000 -v -f tab | ||
| #schema instance flags msize dsize hsize uid gid perm update duration age_seconds age_intervals info |
There was a problem hiding this comment.
Since 'age_seconds' and 'age_intervals' are not self-explanatory, they may benefit from brief description in the man page. What do you think?
There was a problem hiding this comment.
I'll fixed that if i've missed it.
@baallan I gave |
|
@tom95858 @nichamon/the ldms-ug identified incompletenesses in this (formatting of record-list data, details in the man page, user-specified separator instead of just tab). If you want to back out the commits, that's fine. If you just want a second pr to fix the other items, i can open a bug to document the problems and finish the changes i'm already working on to address the feedback. |
|
@nichamon, can you please clarify your comment regarding the json formatting with lists of records. The comment says it formats the list of record data 'correctly.' Did you mean 'incorrectly?' |
This extends (on main) ldms_ls to generate json or tabbed output for machine consumption such as python (json, pandas) and cut or spreadsheets.
In the json and tabbed formats, the age of sets at the last update is included in the outputs triggered by the -v flag. Age is reported in seconds and in intervals (where expected interval hint is available).
The addition of age data is not made to the base pretty-print -v, as this would break existing scripts that derive data from ldms_ls -v.