Increasing test coverage?

[danielinteractive]

Hi @aphalo , first of all thanks for the nice package, really useful extensions here!

One question - we are currently looking into this package (actually only a few of its functions) to use for cases where we need to "validate" the packages. Without going into details, the hard requirement there is to have sufficient unit test coverage of the package.
So currently we see around 20% test coverage in ggpp. But we would need at least 80% overall, and each function tested at least with one test, to get through our requirements.

I was wondering - would you be open to us contributing tests via PRs? Since that seems the most straightforward way.

Cheers
Daniel

[aphalo]

@danielinteractive, you are welcome! Very nice to know that you find the package useful! I am not sure what is the actual coverage of the existing tests, but 20% seems lower than what I would expect. The package has quite a few test cases using testthat but I have been leaving those tests out of the CRAN submissions because they are mainly visual tests that vdiffr very easily reports as not passed even with the visual review showing no changes. Say for example running and saving tests results in the now pending 'ggplot2' 3.4.0-rc results in some 20 test cases failing when switching to 'ggplot2' 3.3.6 and vice-versa, but without any detectable visual change in the output. So, I think that in addition to adding new test cases, a solution to this problem is needed. In other words most of the saved test results in the current tests are specific to 'ggplot2' versions. The latest version of 'ggpp', available here in GitHub, includes 206 test cases using 'testthat' and 'vdiffr'. This is under development and some newer code is still missing tests.

I have for some time been planning to try to submit 'ggpp' and 'ggpmisc' for peer review at ROpenScience but have lacked the time to get these packages ready for such submission. So, pull requests are very welcome for improved test coverage and also exchange of ideas about possible updates to the package code itself, if needed for validation/compliance. I have found that getting good test coverage for graphical output is tricky because updates to 'ggplot2' and 'vdiffr' can change the rendered plot in ways that do not affect how a plot looks like. For some functions like position functions, it would most likely better to create test cases that do not render a plot, and test the actual returned numerical values instead.

One final thought is that the code in 'ggpp' was originally in 'ggpmisc' and some of it may be indirectly covered by test cases in 'ggpmisc'.

Hoping to hear from you soon. (I may not answer straight away as I will be travelling for the next three weeks, but I will have internet access every day.)

[danielinteractive]

Thanks @aphalo for the fast reply! Great. So our baseline number here comes from out-of-the-box covr::package_coverage() on the GitHub source package.
Re: different ggplot2 versions. That is interesting. We could of course skip certain tests based on the version of it.
Unfortunately, vdiffr does not seem to have a variant feature such as the one in shinytest2 which would allow to save different variants for different versions.

[aphalo]

@danielinteractive o.k., yes, I see 23%. Several of the functions with 0% coverage have documentation examples that could be used as tests. However, I have been thinking about how to make the tests more robust after recently reading a blog post by the current maintainer of 'ggplot2' at (https://www.tidyverse.org/blog/2022/09/playing-on-the-same-team-as-your-dependecy/#testing-testing).

[danielinteractive]

Interesting- thanks for sharing. So maybe using layer_data() (and other helper functions, see https://ggplot2.tidyverse.org/reference/ggplot_build.html) to access specific columns resulting from your extensions could be more robust than using vdiffr?

[aphalo]

I explored a bit the tests in the 'ggplot2' 3.4.0-rc branch. I see layer_grobs() used for geoms, while layer_data() seems useful for testing position functions and stats. The tests in the 'ggplot2' release candidate have 70% coverage and some of the tests seem to target specific possible bugs. I think to reach >80% test coverage with effective tests in 'ggpp' we may need to keep some vdiffr-based tests for geoms, unless a good way of comparing gTree (grob tree objects) can be found. So, probably, most reasonable would be when possible not to use vdiffr in new tests, but fall back on them when needed and for the time being to keep existing vdiffr test cases as they are.

I expect to be busy with other tasks during the next 20 days, with little time available for 'ggpp' before 20 November. I will review any pull request as soon as possible but a.s.a.p. may mean a few days delay during this period. I just noticed that 'ggplot2' 3.4.0 is now in the main branch, about to be submitted to CRAN.

[aphalo]

@danielinteractive @sob2021 In branch quadrant-filter I am implementing some new functionality in pre-existing functions. I am also adding test cases to cover these changes. I will merge this branch into master once these test cases are in place.

[aphalo]

@danielinteractive @sob2021 I have added unit tests that should provide nearly 100% coverage for stat_dens1d_labels(). I see with covr::package_coverage() and more clearly with devtools:::test_coverage() that lines of code that are actually tested are being missed by the coverage measurement.

Do you know of a way to get around this problem? Or another way of measuring coverage that is more reliable and at the same time acceptable for your aims.

[danielinteractive]

cool thanks @aphalo ! Unfortunately I am not an expert (yet) with the coverage measurement.
(Are you sure though that the tests are really covering the lines which are missed by the coverage reporting? E.g. did you put in browser() or stop() there and confirm that they are being executed?)
Good news is that we don't need 100% reported coverage. At least 80% is enough for each exported function.

[aphalo]

@danielinteractive I have not tried browser() and stop() yet. But, a quick look shows most if not all code behind else statements being ignored when measuring coverage. Code in some if staments is also not "seen". I also see lines that must have run in a test to produce the plot output, but remain shown in the listing in RStudio's viewer highlighted as not covered. I am not sure that it is like this in every case, but in the tests I added during the weekend I tried to include most possible variations of arguments and their combinations to no avail. I tried with five different stats, in reality they share among them two compute_functions, and the coverage I managed is well below 80% in all cases. I had plenty of unit tests for these functions from earlier and defined quite a few additional ones to get more than 200 tests in all. The coverages reached are: 75%, 73%, 72% and 52% per .R file.

'ggplot2' is tricky to work with as even breakpoints set in the debugger are in most cases ignored, requiring use print() inserted in the code for debugging.

In the 'covr' repo in GitHub and in its vignette there are some dicussions about different difficulties, including dealing with if staements. One thing that could be tried is to run the coverage tests on other architeechtures or with earlier versions of R. I will see if I can set up GitHub action to test coverage.

[aphalo]

@danielinteractive I was able to setup the action easily but some of the 'vdiffr' tests produce plots that seem to be different in Ubuntu at GitHub than on Windows 10 locally. So coverage is not measured. As part of this process I added the test cases to the package build but for the same reason this broke the CHECK Action in Github. I had to push these changes to GitHub to see if they worked. The commints are now reverted and pushed back the version that does not include tests in the package build.

[danielinteractive]

Hi Pedro @aphalo , let us try to have @cicdguy make a PR to introduce covr action in this repo.

Regarding failing vdiffr tests, maybe as a temporary workaround we could insert testthat::skip_on_covr() statements onto the 2 failing tests?

Unfortunately r-lib/vdiffr#125 has not been started yet, this would be the final solution, because then you could store snapshots for different OS. (Maybe you would like to comment there to support the proposal)

[aphalo]

@danielinteractive The coverage workflow is now working, and the package build now includes all tests so that the CHECK workflow also runs them. BFalquet's pull request #23 still needs improvements. alexjoerich's pull request #24 looks good to me, although testing with different summary functions will not increase much the coverage. It is still marked as draft but I reviewed it giving some suggestions. It should now be possible to see what contibution these tests make to coverage. I am not sure how you are organizing this, are you supervising these people's work/training? Would you like to have additional rights to the repo?

[danielinteractive]

thanks @aphalo that is great! Yeah we coordinate this here in our team. We will continue pushing on the mentioned PRs and ping you for re-review once ready. I don't think more rights are needed at this point

[aphalo]

@danielinteractive Hi Daniel. I just watched "Statistical Software Engineering 101: Unit Testing for R Developers - The Basics". Very useful and clear! Although, the unit tests in my packages have had relatively low coverage, tests have indeed been very useful. 'ggpp' had specially low coverage, because the geoms where earlier part of 'ggpmisc' and jointly tested with the stats in the unit tests that are now separated in 'ggpmisc', I have been lazy and run locally these tests indirectly with 'revdep'. The usefulness of tests has been most noticeable in 'photobiology' a package much larger and complex than 'ggpp' defining an S3 class hierarchy and numerous methods and operators for the classes. It currently has nearly 5000 unit tests but still only 51% coverage, this is not as bad as it looks as the most crucial things are being tested thoroughly. One thing that you do not mention in the video is that in my experience the unit tests, if included in the package as they are in 'photobiology' and now also in 'ggpp' ensure that changes in the dependencies are detected early enough, even by the maintainers of the dependencies before the updates to their packages are accepted in CRAN. The other thing is that visual tests can be a pain, because of false positives that need to be checked one by one visually. If one has hundreds of them as in 'ggpmisc' it can take quite some time to check them. Some updates to 'ggplot2' or 'vdiffr' have made all tests fail, forcing me to walk one by one through unit tests even when nothing was actually broken (things like very small changes in the thickness of lines used, even visually, undetectable or changes to the default justification of the plot title). So, I think, that for non-visual tests the cost once written is close to zero, while visual tests do add a cost to maintenance that needs to be counterbalanced by a greater benefit. Nowadays, we have the tools to avoid the use of visual tests as I have learnt from your group's contribution to 'ggpp'. I appreciate very much these contributions. Please, feel free to add Dinakar and BFalquet as contributors if you think their contributions are signicant enough for this.

[aphalo]

@danielinteractive @cicdguy One problem that is keeping the "apparent" coverage low is that compute_panel and compute_group functions are defined separately and then assigned when the ggproto object is built. If the anonymous functions are assigned directly, 'covr' sees them, but if instead the functions are named and assigned using the name 'covr' does not see them. I edited this in a few cases and I will edit the other stats and geoms. However, in a few cases the same functions are used in more than one stat, so defining them inline as anonymous would mean duplicating quite a few lines of code. Do you have any idea on how to get around this?

[danielinteractive]

Thanks @aphalo for your feedback on the unit testing video - actually we are about to record a second "advanced" one (together with @yonicd), so I will try to include your suggested topics in there.

About the function renaming - yeah not sure if there is a general workaround for this, maybe it could be a good question to the covr authors? I wonder if the easiest here would be then to just repeat/duplicate the tests for the named functions which are used in more than one stat?

[aphalo]

@danielinteractive I thought about testing those functions on their own but then I would need to fake the arguments that the function will get as input and then test the returned grob. I will just duplicate the code adding comments to get the needed coverage and in parallel discuss the issue with the authors of 'covr'. This will allow us to see where more tests are still needed. Much of the low coverage is due to this as most of the needed tests are already in place but not seen when the coverage percent is computed.

[aphalo]

@danielinteractive Now the overall coverage is about 75%. I am almost sure all tests are now seen by 'covr' so in some functions there is still code that is not being tested.

[danielinteractive]

That number sounds very nice already, thanks Pedro for your help with this! We are currently finishing up a few items we had on the radar for tests, and will then check back with our validation experts whether we pass the requirements already, or whether we need to add more tests.

[aphalo]

@danielinteractive Now the overall coverage is about 88%. I was wrong, there was still code not seen. I also added tests for some internal functions as I promised to do some time ago. We are lacking tests for one statistic with 0% coverage, otherwise all files report > 75% coverage. An issue was raised yesterday about a bug in a boundary case with the density stats with too few observations, but I fixed it.

[danielinteractive]

Awesome! So we should be there very soon now.

[yonicd]

Hi @aphalo

I collaborate with @danielinteractive on a few repos and wanted to suggest a package called {covrpage}. You may find helpful to track and communicate the testing you are developing.

It can be used within the regular CI/CD framework to generate the report as part of the regular commit/PR process.

From running it on the master branch, I am seeing that a few tests are not passing and two files that dont have any tests running on them

• R/position-nudge-line.R
• R/stat-format-table.R

Happy to help set up the automated report for you if you find it useful.

Yoni

Expand to see the full report

Coverage

Coverage summary is created using the
covr package.

## ⚠️ Not All Tests Passed
##   Coverage statistics are approximations of the non-failing tests.
##   Use with caution
## 
##  For further investigation check in testthat summary tables.

Object	Coverage (%)
ggpp	88.91
R/position-nudge-line.R	0.00
R/stat-format-table.R	0.00
R/annotate.r	75.93
R/position-nudge-dodge2.R	76.92
R/position-nudge-stack.R	81.03
R/try-data-frame.R	81.25
R/stat-dens2d-filter.r	81.82
R/geom-label-linked.r	83.33
R/geom-margin-grob.r	84.21
R/geom-text-npc.r	84.21
R/position-nudge-dodge.R	86.11
R/position-nudge-jitter.R	87.50
R/geom-text-linked.r	87.88
R/position-nudge-center.R	90.00
R/dark-or-light.R	91.67
R/geom-quadrant-lines.R	93.62
R/stat-apply.R	93.65
R/geom-point-linked.r	94.17
R/geom-table.R	94.71
R/geom-label-npc.r	95.45
R/stat-dens2d-labels.r	95.65
R/stat-quadrant-counts.R	96.00
R/compute-npc.r	100.00
R/geom-grob.R	100.00
R/geom-margin-arrow.r	100.00
R/geom-margin-point.r	100.00
R/geom-plot.R	100.00
R/position-nudge-to.R	100.00
R/scale-continuous-npc.r	100.00
R/stat-dens1d-filter.r	100.00
R/stat-dens1d-labels.r	100.00
R/stat-panel-counts.R	100.00
R/utilities.R	100.00

Unit Tests

Unit Test summary is created using the
testthat package.

file	n	time	error	failed	skipped	warning	icon
test-annotate.R	12	2.410	0	2	0	0	🛑
test-compute_npcx.R	10	0.013	0	0	0	0
test-compute_npcy.R	10	0.011	0	0	0	0
test-dark_or_light.R	14	0.065	0	0	0	0
test-geom_grob.R	14	0.168	0	0	0	0
test-geom_label_npc.R	2	0.015	0	0	0	0
test-geom_label_s.R	2	0.016	0	0	0	0
test-geom_point_s.R	9	0.048	0	0	0	0
test-geom_x_margin_arrow.R	4	0.130	0	0	0	0
test-geom_x_margin_grob.R	1	0.264	0	0	0	0
test-geom_x_margin_point.R	4	0.161	0	0	0	0
test-geom_y_margin_arrow.R	6	0.123	0	0	0	0
test-geom_y_margin_grob.R	1	0.137	0	0	0	0
test-geom_y_margin_point.R	6	0.129	0	0	0	0
test-geom-label.R	4	0.474	0	0	0	0
test-geom-plot.R	7	1.357	0	0	0	0
test-geom-quadrant-lines.R	6	0.438	0	0	0	0
test-geom-table.R	32	6.079	0	7	0	0	🛑
test-geom-text_linked.R	43	5.493	0	0	0	0
test-geom-text.R	4	0.444	0	0	0	0
test-ggplot_ts.R	14	0.525	0	0	0	0
test-position_dodge_keep.R	5	0.007	0	0	0	0
test-position_dodge2_keep.R	5	0.005	0	0	0	0
test-position_dodge2nudge.R	8	0.039	0	0	0	0
test-position_fill_keep.R	5	0.006	0	0	0	0
test-position_fillnudge.R	5	0.035	0	0	0	0
test-position_jitter_keep.R	5	0.013	0	0	0	0
test-position_jitternudge.R	9	0.072	0	0	0	0
test-position_stack_keep.R	5	0.005	0	0	0	0
test-position-nudge.R	25	2.713	0	0	1	0	🔶
test-stat_centroid.R	3	0.121	0	0	0	0
test-stat_dens1d_filter_g.R	4	0.027	0	0	0	0
test-stat_dens2d_filter_g.R	3	0.022	0	0	0	0
test-stat_panel_counts.R	19	1.112	0	0	0	0
test-stat_quadrant_counts.R	10	0.444	0	0	0	0
test-stat_summary_xy.R	8	0.201	0	0	0	0
test-stat-d1d-flt.R	158	14.820	0	0	0	0
test-stat-d2d-flt.R	146	11.177	0	0	0	0
test-try_data_frame.R	18	0.056	0	0	0	0
test-try_tibble.R	4	0.018	0	0	0	0
test-ttheme_gtbw.R	22	0.035	0	0	0	0
test-ttheme_gtlight.R	22	0.035	0	0	0	0
test-ttheme_gtminimal.R	4	0.018	0	0	0	0
test-ttheme_gtplain.R	10	0.023	0	0	0	0
test-ttheme_gtsimple.R	22	0.037	0	0	0	0
test-ttheme_gtstripes.R	22	0.053	0	0	0	0
test-utils.R	8	0.012	0	0	0	0

Show Detailed Test Results

file	context	test	status	n	time	icon
test-annotate.R	annotate	dates in segment annotation work	PASS	1	0.142
test-annotate.R	annotate	segment annotations transform with scales	PASS	1	0.124
test-annotate.R	ggpp_annotate	ggpp::annotate works with npc pseudo-aesthetics	FAILED	10	2.144	🛑
test-compute_npcx.R	compute_npcx	character input value returns correct numeric value	PASS	7	0.007
test-compute_npcx.R	compute_npcx	factor input values returns correct numeric value	PASS	1	0.002
test-compute_npcx.R	compute_npcx	numeric input value returns the same value if between 0 and 1	PASS	1	0.002
test-compute_npcx.R	compute_npcx	numeric input value returns 0 if less than 0, returns 1 if greater than 1	PASS	1	0.002
test-compute_npcy.R	compute_npcy	character input value returns correct numeric value	PASS	7	0.008
test-compute_npcy.R	compute_npcy	factor input values returns correct numeric value	PASS	1	0.001
test-compute_npcy.R	compute_npcy	numeric input values returns the same value if between 0 and 1	PASS	1	0.001
test-compute_npcy.R	compute_npcy	numeric input values returns 0 if less than 0, returns 1 if greater than 1	PASS	1	0.001
test-dark_or_light.R	dark_or_light	color: yellow, switch to black, default threshold	PASS	1	0.002
test-dark_or_light.R	dark_or_light	color: darkblue, switch to white, default threshold	PASS	1	0.001
test-dark_or_light.R	dark_or_light	color: #FFFFFF, switch to black, default threshold	PASS	1	0.001
test-dark_or_light.R	dark_or_light	color: #000000, switch to white, default threshold	PASS	1	0.001
test-dark_or_light.R	dark_or_light	color: #000000, switch to light.color specified, default threshold	PASS	1	0.001
test-dark_or_light.R	dark_or_light	color: #000000, switch to dark.color specified, default threshold	PASS	1	0.001
test-dark_or_light.R	dark_or_light	color: yellow, threshold 0.8, switch to white	PASS	1	0.001
test-dark_or_light.R	dark_or_light	threshold: off range triggers error	PASS	4	0.017
test-dark_or_light.R	dark_or_light	threshold: length != 1 triggers error	PASS	3	0.040
test-geom_grob.R	geom_grob	geom_grob pos_or_nudge	PASS	1	0.013
test-geom_grob.R	geom_grob	geom_grob no segment	PASS	4	0.036
test-geom_grob.R	geom_grob	geom_grob segment drawn	PASS	3	0.035
test-geom_grob.R	geom_grob	geom_grob segment disabled	PASS	3	0.034
test-geom_grob.R	geom_grob	geom_grob_npc	PASS	3	0.050
test-geom_label_npc.R	geom_label_npc	geom_label_npc, runs successfully	PASS	1	0.007
test-geom_label_npc.R	geom_label_npc	geom_label_npc pos_or_nudge	PASS	1	0.008
test-geom_label_s.R	geom_label_s	geom_label_s runs successfully	PASS	1	0.005
test-geom_label_s.R	geom_label_s	geom_label_s pos_or_nudge	PASS	1	0.011
test-geom_point_s.R	geom_point_s	geom_point_s runs successfully	PASS	1	0.009
test-geom_point_s.R	geom_point_s	geom_point_s pos_or_nudge	PASS	1	0.013
test-geom_point_s.R	geom_point_s	translate_shape_string maps correctly	PASS	7	0.026
test-geom_x_margin_arrow.R	geom_x_margin_arrow	geom_x_margin_arrow, missing xintercept	PASS	1	0.106
test-geom_x_margin_arrow.R	geom_x_margin_arrow	geom_x_margin_arrow, xintercept single value	PASS	1	0.011
test-geom_x_margin_arrow.R	geom_x_margin_arrow	geom_x_margin_arrow, xintercept dataframe	PASS	2	0.013
test-geom_x_margin_grob.R	geom_x_margin_grob	geom_x_margin_grob, missing xintercept	PASS	1	0.264
test-geom_x_margin_point.R	geom_x_margin_point	geom_x_margin_point, missing xintercept	PASS	1	0.134
test-geom_x_margin_point.R	geom_x_margin_point	geom_x_margin_point, xintercept single value	PASS	1	0.011
test-geom_x_margin_point.R	geom_x_margin_point	geom_x_margin_point, xintercept dataframe	PASS	2	0.016
test-geom_y_margin_arrow.R	geom_y_margin_arrow	geom_y_margin_arrow, missing yintercept	PASS	1	0.091
test-geom_y_margin_arrow.R	geom_y_margin_arrow	geom_y_margin_arrow, yintercept	PASS	3	0.018
test-geom_y_margin_arrow.R	geom_y_margin_arrow	geom_y_margin_arrow, yintercept dataframe	PASS	2	0.014
test-geom_y_margin_grob.R	geom_y_margin_grob	geom_y_margin_grob, missing yintercept	PASS	1	0.137
test-geom_y_margin_point.R	geom_y_margin_point	geom_y_margin_point, missing yintercept	PASS	1	0.094
test-geom_y_margin_point.R	geom_y_margin_point	geom_y_margin_point, yintercept	PASS	3	0.020
test-geom_y_margin_point.R	geom_y_margin_point	geom_y_margin_point, yintercept dataframe	PASS	2	0.015
test-geom-label.R	geom_label_npc	multiple_rows_tb	PASS	4	0.474
test-geom-plot.R	geom_plot	data.frame	PASS	1	0.105
test-geom-plot.R	geom_plot	multiple_ggplot	PASS	2	0.791
test-geom-plot.R	geom_plot	examples_geom_plot	PASS	2	0.407
test-geom-plot.R	geom_plot	nudge_x_geom_plot	PASS	1	0.035
test-geom-plot.R	geom_plot	nudge_x_and_position_geom_plot_fails	PASS	1	0.019
test-geom-quadrant-lines.R	geom_quadrant_lines	geom_quadrant_lines works as expected with default values	PASS	1	0.029
test-geom-quadrant-lines.R	geom_quadrant_lines	geom_quadrant_lines works as expected with custom parameter values	PASS	3	0.251
test-geom-quadrant-lines.R	geom_quadrant_lines	geom_vhlines returns an error if neither x nor y intercept is provided	PASS	1	0.129
test-geom-quadrant-lines.R	geom_quadrant_lines	geom_vhlines works as expected with multiple intercepts	PASS	1	0.029
test-geom-table.R	geom_tb	multiple_rows_tb	FAILED	1	0.700	🛑
test-geom-table.R	geom_tb	numbers_tb	FAILED	6	2.156	🛑
test-geom-table.R	geom_tb	theme_tb	PASS	15	2.332
test-geom-table.R	geom_tb	letters_tb	PASS	1	0.151
test-geom-table.R	geom_tb	parsed_tb	PASS	3	0.413
test-geom-table.R	geom_tb	geom_table_pos_or_nudge	PASS	1	0.016
test-geom-table.R	geom_tb	geom_table_string_left_hjust	PASS	1	0.006
test-geom-table.R	geom_tb	letters_tb_npc	PASS	1	0.108
test-geom-table.R	geom_tb	multiple_rows_tb_npc	PASS	1	0.173
test-geom-table.R	geom_tb	geom_table_npc_string_center_hjust	PASS	2	0.024
test-geom-text_linked.R	geom_text_s	text_linked_just	PASS	27	3.159
test-geom-text_linked.R	geom_text_s	text_linked_size	PASS	1	0.233
test-geom-text_linked.R	geom_text_s	text_linked_linewidth	PASS	3	0.375
test-geom-text_linked.R	geom_text_s	text_linked_color	PASS	12	1.726
test-geom-text.R	geom_text_npc	multiple_rows_tb	PASS	4	0.444
test-ggplot_ts.R	ggplot.ts	ggplot.ts works as expected with the yearly time series.	PASS	3	0.030
test-ggplot_ts.R	ggplot.ts	ggplot.ts works as expected with the monthly time series.	PASS	3	0.089
test-ggplot_ts.R	ggplot.ts	ggplot.ts time resolution works as expected.	PASS	3	0.149
test-ggplot_ts.R	ggplot.ts	ggplot.ts mapping works as expected.	PASS	5	0.257
test-position_dodge_keep.R	position_dodge_keep	test expected arguments	PASS	5	0.007
test-position_dodge2_keep.R	position_dodge2_keep	test expected arguments	PASS	5	0.005
test-position_dodge2nudge.R	position_dodge2nudge	incorrect kept.origin used	PASS	1	0.028
test-position_dodge2nudge.R	position_dodge2nudge	correct kept.origin used	PASS	1	0.002
test-position_dodge2nudge.R	position_dodge2nudge	test if correct arguments are assigned	PASS	6	0.009
test-position_fill_keep.R	position_fill_keep	test vjust, reverse, x, y and kept.origin arguments	PASS	5	0.006
test-position_fillnudge.R	position_fillnudge	incorrect kept.origin used	PASS	1	0.028
test-position_fillnudge.R	position_fillnudge	correct kept.origin used	PASS	1	0.001
test-position_fillnudge.R	position_fillnudge	correct reverse, vjust and x assigned	PASS	3	0.006
test-position_jitter_keep.R	position_jitter_keep	test expected arguments	PASS	5	0.013
test-position_jitternudge.R	position_jitternudge	incorrect kept.origin used	PASS	1	0.028
test-position_jitternudge.R	position_jitternudge	correct kept.origin used	PASS	1	0.002
test-position_jitternudge.R	position_jitternudge	incorrect nudge.from used	PASS	1	0.036
test-position_jitternudge.R	position_jitternudge	correct nudge.from used	PASS	1	0.001
test-position_jitternudge.R	position_jitternudge	test if correct arguments are assigned	PASS	5	0.005
test-position_stack_keep.R	position_stack_keep	test vjust, reverse, x, y and kept.origin arguments	PASS	5	0.005
test-position-nudge.R	positions	stacknudge	PASS	4	0.562
test-position-nudge.R	positions	dodgenudge	PASS	4	0.511
test-position-nudge.R	positions	nudge_keep	PASS	4	0.528
test-position-nudge.R	positions	nudge_center	PASS	7	0.787
test-position-nudge.R	positions	nudge_line	SKIPPED	3	0.061	🔶
test-position-nudge.R	positions	nudge_to	PASS	3	0.264
test-stat_centroid.R	stat_centroid	stat_centroid, using default shape and size	PASS	1	0.037
test-stat_centroid.R	stat_centroid	stat_centroid, changed default shape and size	PASS	1	0.051
test-stat_centroid.R	stat_centroid	stat_centroid, geom ‘rug’ function median applied	PASS	1	0.033
test-stat_dens1d_filter_g.R	stat_dens1d_filter_g	stat_dens1d_filter_g, default arguments	PASS	1	0.005
test-stat_dens1d_filter_g.R	stat_dens1d_filter_g	stat_dens1d_filter_g, change default arguments	PASS	2	0.010
test-stat_dens1d_filter_g.R	stat_dens1d_filter_g	stat_dens1d_filter_g, incorrect argument	PASS	1	0.012
test-stat_dens2d_filter_g.R	stat_dens2d_filter_g	stat_dens2d_filter_g, default arguments	PASS	1	0.005
test-stat_dens2d_filter_g.R	stat_dens2d_filter_g	stat_dens2d_filter_g, change default arguments	PASS	1	0.006
test-stat_dens2d_filter_g.R	stat_dens2d_filter_g	stat_dens2d_filter_g, incorrect argument	PASS	1	0.011
test-stat_panel_counts.R	stat_panel_counts	stat_panel_counts	PASS	8	0.462
test-stat_panel_counts.R	stat_panel_counts	stat_group_counts	PASS	11	0.650
test-stat_quadrant_counts.R	stat_quadrant_counts	stat_quadrant_counts	PASS	10	0.444
test-stat_summary_xy.R	stat_summary_xy	stat_summary_xy	PASS	6	0.160
test-stat_summary_xy.R	stat_summary_xy	stat_apply_group	PASS	2	0.041
test-stat-d1d-flt.R	stat_dens1d_filter	filter_x_params	PASS	31	0.231
test-stat-d1d-flt.R	stat_dens1d_filter	numbers_x_tb	PASS	22	3.141
test-stat-d1d-flt.R	stat_dens1d_filter	numbers_y_tb	PASS	15	2.192
test-stat-d1d-flt.R	stat_dens1d_filter	labels_x_params	PASS	39	2.168
test-stat-d1d-flt.R	stat_dens1d_filter	labels_y_params	PASS	17	2.056
test-stat-d1d-flt.R	stat_dens1d_filter	labels_x_tb	PASS	17	2.506
test-stat-d1d-flt.R	stat_dens1d_filter	labels_y_tb	PASS	17	2.526
test-stat-d2d-flt.R	stat_dens2d_filter	filter_params	PASS	43	0.285
test-stat-d2d-flt.R	stat_dens2d_filter	numbers_tb	PASS	19	4.657
test-stat-d2d-flt.R	stat_dens2d_filter	labels_params	PASS	35	2.064
test-stat-d2d-flt.R	stat_dens2d_filter	labels_tb	PASS	15	4.114
test-stat-d2d-flt.R	stat_dens2d_filter	these2logicalarguments with label	PASS	18	0.034
test-stat-d2d-flt.R	stat_dens2d_filter	these2logicalarguments without label	PASS	16	0.023
test-try_data_frame.R	try_data_frame	try_data_frame, compare output	PASS	1	0.004
test-try_data_frame.R	try_data_frame	try_data_frame, check if dataframe	PASS	1	0.004
test-try_data_frame.R	try_data_frame	try_data_frame, check format of time	PASS	2	0.022
test-try_data_frame.R	try_data_frame	try_data_frame, check format of time with xts	PASS	2	0.007
test-try_data_frame.R	try_data_frame	try_data_frame, if data frame pass through	PASS	2	0.003
test-try_data_frame.R	try_data_frame	try_data_frame, if list to data frame	PASS	2	0.002
test-try_data_frame.R	try_data_frame	ggplot.ts, if list to data frame num	PASS	4	0.007
test-try_data_frame.R	try_data_frame	ggplot.ts, if list to data frame date	PASS	4	0.007
test-try_tibble.R	try_tibble	try_tibble, compare output	PASS	1	0.005
test-try_tibble.R	try_tibble	try_tibble, check class	PASS	1	0.004
test-try_tibble.R	try_tibble	try_tibble, check format of time	PASS	2	0.009
test-ttheme_gtbw.R	ttheme_gtbw	ttheme_gtbw, default text elements	PASS	11	0.018
test-ttheme_gtbw.R	ttheme_gtbw	ttheme_gtbw, check changes to text elements	PASS	11	0.017
test-ttheme_gtlight.R	ttheme_gtlight	ttheme_gtlight, default text elements	PASS	11	0.018
test-ttheme_gtlight.R	ttheme_gtlight	ttheme_gtlight, check changes to text elements	PASS	11	0.017
test-ttheme_gtminimal.R	ttheme_gtminimal	ttheme_gtminimal, default text elements	PASS	2	0.009
test-ttheme_gtminimal.R	ttheme_gtminimal	ttheme_gtminimal, check changes to text elements	PASS	2	0.009
test-ttheme_gtplain.R	ttheme_gtplain	ttheme_gtplain, default text elements	PASS	5	0.012
test-ttheme_gtplain.R	ttheme_gtplain	ttheme_gtplain, check changes to text elements	PASS	5	0.011
test-ttheme_gtsimple.R	ttheme_gtsimple	ttheme_gtsimple, default text elements	PASS	11	0.019
test-ttheme_gtsimple.R	ttheme_gtsimple	ttheme_gtsimple, check changes to text elements	PASS	11	0.018
test-ttheme_gtstripes.R	ttheme_gtstripes	ttheme_gtstripes, default text elements	PASS	11	0.035
test-ttheme_gtstripes.R	ttheme_gtstripes	ttheme_gtstripes, check changes to text elements	PASS	11	0.018
test-utils.R	utils	parse_safe works with simple expressions	PASS	4	0.006
test-utils.R	utils	parse_safe works with multi expressions	PASS	4	0.006

Failed	Warning	Skipped
🛑	⚠️	🔶

Session Info

Field	Value
Version	R version 4.2.1 (2022-06-23)
Platform	x86_64-apple-darwin17.0 (64-bit)
Running	macOS Big Sur 11.7.2
Language	en_US
Timezone	America/New_York

Package	Version
testthat	3.1.6
covr	3.6.1
covrpage	0.2

[aphalo]

Hi @yonicd
Many thanks! I looked at your repo for 'covrpage'. 'covrpage' does look very useful. I have been using the 'devtools' based report table for coverage in RStudio to find which lines of code are not covered by tests. However, 'covrpage' adds the tests' outcome and lists them, so surely it would be very useful to add the automated report.

Overall, though, the main issue with test coverage in 'ggpp' is that because of how 'ggplot2' works with a lot of code as functions assigned as list members, 'covr' only "sees" a fraction of all the code in the package, so also the test coverage % is not computed based on all the code in the package nor on all the code that tests are testing.

I did not setup the covr action myself, Danikar, a member of Daniel's team did. I would greatly appreciate your help with setting up the 'covrpage' action. I am quite busy at the moment.

Pedro.

[aphalo]

Hi @yonicd
I am planning to follow as much as possible ROpenScience recomendations. One point they have added a couple of years ago is: "2.2.1 Travis CI (Linux and Mac OSX) We recommend moving away from Travis.

By looking at 'covrpage' you seem to be still using Travis. I am trying to only use GitHub actions for CI. I guess the coverage table can be also generated locally... Is this true?

[aphalo]

@yonicd @danielinteractive In the case of 'vdiffr' tests, for the same reason that they are disabled on CRAN, it would be unwise an confussing to have them in a vignette. In a README they would need a very clear warning. These tests break much more often than "normal" tests, and most failures are not because of bugs but because of irrelevant changes, usually invisible changes, in the rendered SVG output caused by intentional updates in depedencies, including even system fonts, etc.

The test failures you found, seem to be an example of this. All tests pass cleanly locally and also on GitHub CI but not under the setup you used. I think a precondition for publishing test results, as oppossed to using them internally, is that false positives are a rarity rather than the most frequent cause of test failures. I am already uneasy about a test coverage badge that is based on a coverage that ignores all anonymous functions saved as part of lists, which are at the core of plot layer functions.

[aphalo]

@yonicd @danielinteractive I merged the pull request from @alexjoerich. The test coverage action that Danikar setup is no longer working, possibly because of the recent commits at https://github.com/insightsengineering/
r.pkg.template.

[cicdguy]

@aphalo - apologies for that. Didn't realize ggpp doesn't leverage gh-pages to host the package's documentation. I've proposed a fix here to run the coverage reports: #40

[aphalo]

'ggpp' 0.5.2 is in CRAN
@danielinteractive @maksymiuks @cicdguy @yonicd and all the rest of your people: thanks a lot for all your help in improving the package!!

[aphalo]

@danielinteractive I updated stat_quadrant_counts() and stat_groups_counts() and the corresponding unit tests. The default plots remain unchanged, but now labels expressed as a percent or a fraction of the total number of observations are available in addition to label for raw counts of observations. I am letting you know in case you want to have a look. If I find time I will add a new stat before submitting the next version to CRAN (not later than in a couple of weeks, I hope).

[aphalo]

@danielinteractive In a separate branch I am implementing stat_functions(). Once I could phatom a reasonable user interface, coding the bare bones of the stat was very simple. I will add unit tests before merging this branch into master.

[danielinteractive]

Thanks @aphalo for letting me know - great to see that you are adding further nice features to the package. And since you add the unit tests and coverage stays very good, all fine from our perspective :)

[aphalo]

@danielinteractive 'ggpp' 0.5.5 submitted minutes ago to CRAN. Main change is compatibility with 'ggplot2' development (future 3.5.x). At the moment the test action at GitHub is failing. The problem is that there are small differences in the graphical output generated under different versions of R. So, vdiffr tests either fail under R-devel or R-release. These tests are anyway skipped on CRAN. So status at CRAN is OK.

[danielinteractive]

Thanks @aphalo , in those cases I usually regenerate the snapshots so that GitHub actions can pass again.

[aphalo]

@danielinteractive, do you know if it is possible to set-up tests so that different snapshots are used for r-devel, and r-release and r-old-release? The current snapshots are for r-devel, which seems to me the most likely test to break in such a way that would need action on my part to fix it. An alternative would be to use only the results from tests under r-devel to build the icon displayed in the README...
The tests fail because of trivial things, like minor changes in text justification or very minor changes in the borders of labels.

[danielinteractive]

I don't think that is possible. But you could double check with the vdiffr GitHub folks.

[aphalo]

@danielinteractive I am planning to submit 'ggpp' 0.5.7 to CRAN wihtin a couple of days. Coverage is now back to about 93%.