Unexpected behaviour when feeding getLine input to a process

[AliceRixte]

First of all, thank you for this library and the great work !

I've posted this question on stackoverflow, and it looks like nobody there knows the answer, so I'll post it here as an issue. Here is a copy/paste from stack overflow :

I'm trying to use streamly-process to communicate with some REPL in background. It could be Python or anything but here I try to run GHCi. I came up with the following code :

import Data.Word
import qualified Streamly.Data.Stream.Prelude as Stream
import qualified Streamly.Data.Fold.Prelude as Fold
import qualified Streamly.System.Process as Process
import qualified Streamly.Console.Stdio as Stdio
import qualified Streamly.Data.Array.Foreign as Array


stringToByteArray :: String -> Array.Array Word8
stringToByteArray = Array.fromList . map (fromIntegral . fromEnum)

-- a version of getLine that does not ignore the newline character.
getNewLn :: IO String
getNewLn =  fmap (++ "\n") getLine


main :: IO ()
main = do
    Stream.fold (Fold.takeEndBy ( == stringToByteArray "Leaving GHCi.") Stdio.writeChunks) $
        Process.pipeChunks "ghci" [] $
          fmap stringToByteArray $
            Stream.repeatM getNewLn
            -- When using Stream.fromList, it works ! To do so, uncomment the following :

            -- Stream.fromList ["putStrLn \"This works ! \"\n", " 3 + 4\n"]

When I use fromList, it works as expected and returns :

GHCi, version 9.8.2: https://www.haskell.org/ghc/  :? for help
ghci> This works ! 
ghci> 7
ghci> Leaving GHCi.

However, when I'm using getNewLn this is what I get :

GHCi, version 9.8.2: https://www.haskell.org/ghc/  :? for help
ghci> I'm typing
the return key
several times
but nothing happens ...

What did I get wrong ?

[harendra-kumar]

@AliceRixte this looks like because of IO buffering occurring in the pipeChunks API. Let me check and update.

[harendra-kumar]

Try out this fix: #86 . If this works we can add a config option to enable different types of buffering optionally.

[AliceRixte]

Yes it works great ! This is awesome ! I'm amazed at how concise this is :-)

[AliceRixte]

I've been playing a bit more with this and I think it would be great to also have the possibility to use line buffers in the output as well.

For instance, what I'm currently trying to achieve is to make a REPL for an EDSL I made in Haskell, and I'd like to be able to replace the ghci> prompt with my own language prompt. However for now the output chunks of GHCi are a bit all over the place and I don't see an easy way to do this, except line buffering for the output of pipeChunks.

Another example of why this would be useful is the (admitedly not great) hack I came up with to quit as soon as GHCi quit using

Fold.takeEndBy ( == stringToByteArray "Leaving GHCi.") Stdio.writeChunks

This does not work for now because the buffering splits "Leaving GHCi." into 2 chunks.

[harendra-kumar]

If we do line buffering then you cannot output a line until newline is encountered, since ghci prompt does not end with a newline you wont be able to display the prompt correctly.

Not very efficient, but you can output character-by-character for now like this.

import qualified Streamly.Data.Array as Array -- Streamly.Data.Array.Foreign is deprecated
import qualified Streamly.Internal.Data.Fold as Fold -- takeEndBySeq is not exposed yet

    hSetBuffering stdout NoBuffering

    let f =
                Fold.takeEndBySeq (stringToByteArray "Leaving GHCi.")
              $ Fold.drainMapM (putChar . chr . fromIntegral)

    Stream.fold f
        $ Stream.unfoldMany Array.reader
        $ Process.pipeChunks "ghci" []
        $ Stdio.readChunks

We can do it more efficiently in chunks as well, but that will require a bit more complicated code.

BTW, using "Leaving GHCi" to detect the end is hacky, if the ghci output has this string elsewhere e.g. the user types it that will end the session.

[harendra-kumar]

The simplest is to not detect the end by a string, just use end-of-stream to end the session, something like this should work:

import qualified Streamly.Internal.Console.Stdio as Stdio

      Stdio.putChunks
        $ Process.pipeChunks "ghci" []
        $ Stdio.readChunks

[AliceRixte]

Thank you for your insights !

The second code doesn't end the stream on quitting GHC, but it really nice ! However I will process the input line by line before feeding it to GHCi so I think I'll stick to getLine.

For quitting, I think I will use something in the flavor of your AcidRain example, and hijack the ":q" command to change the program state, send ":q" to GHCi and then quit. This will be less hacky. I will need a state anyway for other purposes.

Thanks again for all the support !

[harendra-kumar]

Line by line processing is easy. You can split the input into lines using the stream APIs. In the following example each chunk being sent to pipeChunks is a line:

        Stdio.putChunks
        $ Process.pipeChunks "ghci" []
        $ Stream.foldMany (Fold.takeEndBy (== 10) Array.create)
        $ Stdio.read

[AliceRixte]

Oh this is really cool. I tried to use the same idea to process the output of GHCi line by line, the idea being that I can put my own prompt "myLanguage>", and delete "ghci>" at the begining of every line of the output of GHCi. I suppose I'll use the Parser functionalities to do so.

However if I try to convert back the output of pipeChunks to a stream of Word8 with the following code

Stream.fold Stdio.write
        $ Stream.concatMap Array.read
        $ Process.pipeChunks "ghci" []
        $ Stream.foldMany (Fold.takeEndBy (== 10) Array.create)
        $ Stdio.read

the output is a bit all over the place :

hi

<interactive>:1:1: error: [GHC-88464]
    Variable not in scope: hi
    Suggested fix: Perhaps use `pi' (imported from Prelude)
world !              

<interactive>:2:8: error: [GHC-58481]
    parse error (possibly incorrect indentation or mismatched brackets)
:q
GHCi, version 9.8.2: https://www.haskell.org/ghc/  :? for help
ghci> ghci> ghci> Leaving GHCi.

Notice how GHCi's errors are displayed in the right place and how everything else is flushed only when quitting the program.