Idea for Pandoc Custom Reader

[Abhi-1U]

Based on the custom reader posted as a solution to the test case, I have modified it and tried to pass through non-sweave components (LaTeX) into the pandoc LaTeX reader. However there are a few Issues in properly capturing the non code block parts of the document, which will require some efforts or collaboration to find out what the correct LPEG expression would be to capture it.

-- For better performance we put these functions in local variables:
local P, S, R, Cf, Cc, Ct, V, Cs, Cg, Cb, B, C, Cmt, Cp =
  lpeg.P, lpeg.S, lpeg.R, lpeg.Cf, lpeg.Cc, lpeg.Ct, lpeg.V,
  lpeg.Cs, lpeg.Cg, lpeg.Cb, lpeg.B, lpeg.C, lpeg.Cmt, lpeg.Cp

local whitespacechar = S(" \t\r\n")
local wordchar = (1 - whitespacechar)
local spacechar = S(" \t")
local newline = P"\r"^-1 * P"\n"
local blankline = spacechar^0 * newline
local codeblockstart = P"<<"
                        * spacechar^0 -- Ignore spaces
                        * C((P(1) - P(spacechar^0 * P">>="))^0) -- Capture attributes between << and >>=
                        * spacechar^0 -- Ignore spaces
                        * ">>="
local enddocument = P"\\z"

-- Grammar
G = P{ "Pandoc",
    Pandoc = Ct(V"Block"^0) / pandoc.Pandoc;
    Block = blankline^0 * ( V"CodeBlock" + V"NonCodeBlock");
    CodeBlock = codeblockstart
                * blankline
                * C((1 - (newline * P"@"))^0) -- Capture codes between << >>= and @
                * newline
                * P"@"
                / function(attributes, code) -- Return a CodeBlock with the captured attributes and codes
                    return pandoc.CodeBlock(code, {"r", attributes})
                  end;
    NonCodeBlock = --P(1 - V"CodeBlock")^1 -- A faulty Expression to capture parts of doc other than codeblock (needs work) 
              P(1)^1   -- captures everything (currently works as a proof of concept) / CodeBlock is Ignored in this case.
              / function(text)
              -- debug steps to get the capture contents into a file
              file = io.open("Capture.txt", "w") 
              file:write(text)
              file:close()
              -- treat the captured text as normal LaTeX
              local doc = pandoc.read(text,"latex")
              return pandoc.Para(pandoc.utils.blocks_to_inlines(doc.blocks)) 
             end;  
}


function Reader(input, reader_options)
    return lpeg.match(G, tostring(input))
end

Figuring out the Capture expression will be crucial on working with this approach.

pandoc command

pandoc file.Rnw -f sweave_code_reader.lua -t markdown -o file.md

Test file source

\documentclass{article}
\usepackage{amsmath}
\begin{document}

When the option OutDec is not \texttt{.}, put numbers in \texttt{\textbackslash{}text}. See \#348.

<<>>=
options(OutDec = ',')
@

This is the first test. abc \Sexpr{0.6} def

another test $a = \Sexpr{0.6}$.

and the last one $a = \Sexpr{'0.6'}$.

<<>>=
options(OutDec = '.')
@

This is the first test. abc \Sexpr{0.6} def

another test $a = \Sexpr{0.6}$.

and the last one $a = \Sexpr{'0.6'}$.

\end{document}

[fzyxh]

I created texor-playground repo to show a reproducible test process, this test example has been added to the repo.

[fzyxh]

I create pandoc reader Sweave_code_remove.lua for the rest part of Sweave file, it works together with Sweave_code_reader.lua to separate the .Rnw file into two parts: code chunks and the rest with placeholders.

The output of the example is file.md and file.tex :

``` {r }
options(OutDec = ',')
```

``` {r }
options(OutDec = '.')
```

\documentclass{article}
\usepackage{amsmath}
\begin{document}

When the option OutDec is not \texttt{.}, put numbers in \texttt{\textbackslash{}text}. See \#348.

% R_CODE_CHUNK_PLACEHOLDER

This is the first test. abc \Sexpr{0.6} def

another test $a = \Sexpr{0.6}$.

and the last one $a = \Sexpr{'0.6'}$.

% R_CODE_CHUNK_PLACEHOLDER

This is the first test. abc \Sexpr{0.6} def

another test $a = \Sexpr{0.6}$.

and the last one $a = \Sexpr{'0.6'}$.

\end{document}

To separate the .Rnw file, execute following commands:

pandoc 008-outdec.Rnw --from latex -f sweave_code_reader.lua --lua-filter code_chunk_patch.lua -t markdown-simple_tables-pipe_tables-fenced_code_attributes -o file.md
pandoc 008-outdec.Rnw --from latex -f Sweave_code_remove.lua -t latex -o file.tex

[Abhi-1U]

I played around a bit with your approach, it seems to be good but has a few issues in my mind.

1. The Placeholder is a LaTeX comment, so if we convert it to markdown using texor/pandoc then the comment would be removed from the final file. To preserve it we need to insert the Placeholder as a verbatim/CodeBlock chunk with the placeholder comment.

2. To merge the code chunks into the placeholders, we will need to number them somewhere so we can easily refer to it both in placeholder and the r code chunk file.

3. You can try to use one of the Lua filters I have a written and modify it to read individual code chunks and replace the placeholders.

4. we can also replace the \Sexpr in the math environment by using gsub creatively, You can refer this filter.

5. Lastly the attributes in the code chunk is not properly formatted in the end, one way could be to use a RawBlock instead of CodeBlock to generate the code chunks and place the attributes correctly.

[fzyxh]

Let me reply one by one.

1. Sure we can use any other text to replace code chunks, the comment is just an example.

2. Since the code chunks are extracted sequentially, it doesn't seem to be a need for numbering?

3. Yeah, as \newcommand{} works not well when identifying inline code within inline math like $a = \Sexpr{3 + 3}$, it's a good choice to use gsub. Though we may need to ensure \Sexpr should not be replaced in some cases. As I tested in some examples, \Sexpr works even in verbatim env:

\documentclass{article}
\begin{document}
\SweaveOpts{concordance=TRUE}

This is all you need to do if you want to go back to the Sweave style:

<<setup, include=FALSE>>=
library(knitr)
render_sweave()
@

The quick brown fox jumps over the lazy dog the quick brown fox jumps over the
lazy dog the quick brown fox jumps over the lazy dog.

\begin{verbatim}
try this:
\Sexpr{1+1}
\end{verbatim}

\end{document}

In this case, \Sexpr{1+1} will be shown as 2 after compiling to pdf format. Therefore, maybe all \Sexpr should be regarded as inline code.

4. I'm not clear what the attributes in the code chunk is not properly formatted means. Can you describe it with more detail?

[Abhi-1U]

4. I'm not clear what the attributes in the code chunk is not properly formatted means. Can you describe it with more detail?

You can disregard the last point, after some investigating it turned out to be accepted by the knitting tool.

2. Since the code chunks are extracted sequentially, it doesn't seem to be a need for numbering?

It depends on the method you use to merge the two documents, if it works out without numbering that should be fine as well.

As I tested in some examples, \Sexpr works even in verbatim env

I tried out something like this in R markdown file and it worked as intended :

```
try this
`r 1+1`
```

But the \newcommand{} macro wont replace \Sexpr{} in verbatim environments. So in this case just like math we may have to use gsub possibly.

Therefore, maybe all \Sexpr should be regarded as inline code.

It is regarded as the inline equivalent, however pandoc does not seem to touch the macros inside verbatim/math environments, hence the command is not substituted.

Its good to see we are progressing in the right direction.