Refactor the incremental compiler (#762)

The motivation for this refactoring is to be able to implement language
service support for notebook cells. (You can preview how that's going to
fit together at #759.) But it achieves more than that, by eliminating a
lot of code duplicated between the batch compilation and incremental
compilation paths, devising a coherent model for the incremental
compiler, and clearly separating the evaluation parts of the
`Interpreter` from the compilation parts.

**1. New top-level nodes in the AST and HIR**

Previously, the incremental compilation was made up of loose AST and HIR
nodes stored in `stateful::Interpreter`, and an empty `CompileUnit` in
the package store. This packages in this `CompileUnit` were not updated
as fragments got compiled in.

Every feature in the language service is implemented using an AST
visitor and uses the HIR package for lookups. This imposes the
requirement that notebook cells are able to be represented as AST and
HIR packages. They should be kept up to date with each cell added to the
compilation.

Q# notebook cells were previously thought of as "fragments" (not "real"
packages). This PR makes fragments syntax official, so to speak, by
adding the ability for `ast::Package` and `hir::Package` to contain
top-level statements.

When we are able to represent each incremental unit (notebook cell) as a
whole package, we can eliminate a lot of lowering/resolution/checking
code that was designed to work with loose statement/item nodes (e.g.
`check_stmt_fragment`).. We can now use their package counterparts (e.g.
`check_package`), making for more code shared between the batch and
incremental compilers.

**2. Layering and `qsc::incremental`**

Previously, the incremental compiler implementation was spread out
between `stateful` and `qsc_frontend::incremental`. The
`stateful::Interpret` struct contained too much state, making it really
hard to write code that mutates logically separate parts of the struct
(e.g. compiler state vs. evaluator state) without angering the Rust
borrow checker.

The new `qsc::incremental` module is mainly the compiler-y parts of
`stateful` pulled out into their own module, bridging the gap between
`stateful` and `qsc_frontend::incremental`. The layering is meant to
parallel that of batch compilation:

`qsi` -> `qsc::interpret::stateful` -> `qsc::incremental` ->
`qsc_frontend::incremental`
`qsc` -> `qsc::compiler` -> `qsc_frontend::compiler`

**3. Mutable `CompileUnit` in the `PackageStore`**

#675 required that the current compilation be available to be looked up
in the `PackageStore`, since errors may contain the `PackageId` of the
current package. But the current compilation also needs to be updated
with incremental changes. Adding a `get_mut` to the `PackageStore` felt
wrong since almost all use cases of the `PackageStore` require packages
to remain immutable. To address the awkwardness, the concept of an
`OpenPackageStore` is introduced. A `PackageStore` can be "opened" for
modifications, making it possible to obtain a mutable reference to only
the last `CompileUnit` in the store (the open package). The incremental
compiler uses the open package store to keep updating the compilation,
while the rest of the components (evaluator, error span lookups, etc)
have access to the regular, immutable, `PackageStore` for lookups.

compiler/qsc/src/incremental.rs

@@ -0,0 +1,191 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+use crate::compile::{self, compile, core, std};
+use miette::Diagnostic;
+use qsc_frontend::{
+    compile::{OpenPackageStore, PackageStore, SourceMap, TargetProfile},
+    error::WithSource,
+    incremental::Increment,
+};
+use qsc_hir::hir::PackageId;
+use qsc_passes::{PackageType, PassContext};
+
+/// An incremental Q# compiler.
+pub struct Compiler {
+    /// A package store that contains the current, mutable, `CompileUnit`
+    /// as well as all its immutable dependencies.
+    store: OpenPackageStore,
+    /// The ID of the source package. The source package
+    /// is made up of the initial sources passed in when creating the compiler.
+    source_package_id: PackageId,
+    /// Context for passes that is reused across incremental compilations.
+    passes: PassContext,
+    /// The frontend incremental compiler.
+    frontend: qsc_frontend::incremental::Compiler,
+}
+
+/// An incremental compiler error.
+pub type Errors = Vec<WithSource<compile::Error>>;
+
+impl Compiler {
+    /// Creates a new incremental compiler, compiling the passed in sources.
+    /// # Errors
+    /// If compiling the sources fails, compiler errors are returned.
+    pub fn new(
+        include_std: bool,
+        sources: SourceMap,
+        package_type: PackageType,
+        target: TargetProfile,
+    ) -> Result<Self, Errors> {
+        let core = core();
+        let mut store = PackageStore::new(core);
+        let mut dependencies = Vec::new();
+        if include_std {
+            let std = std(&store, target);
+            let id = store.insert(std);
+            dependencies.push(id);
+        }
+
+        let (unit, errors) = compile(&store, &dependencies, sources, package_type, target);
+        if !errors.is_empty() {
+            return Err(into_errors_with_source(errors, &unit.sources));
+        }
+
+        let source_package_id = store.insert(unit);
+        dependencies.push(source_package_id);
+
+        let frontend = qsc_frontend::incremental::Compiler::new(&store, dependencies, target);
+        let store = store.open();
+
+        Ok(Self {
+            store,
+            source_package_id,
+            frontend,
+            passes: PassContext::new(target),
+        })
+    }
+
+    /// Compiles Q# fragments. Fragments are Q# code that can contain
+    /// top-level statements as well as namespaces. A notebook cell
+    /// or an interpreter entry is an example of fragments.
+    ///
+    /// This method returns the AST and HIR packages that were created as a result of
+    /// the compilation, however it does *not* update the current compilation.
+    ///
+    /// The caller can use the returned packages to perform passes,
+    /// get information about the newly added items, or do other modifications.
+    /// It is then the caller's responsibility to merge
+    /// these packages into the current `CompileUnit` using the `update()` method.
+    pub fn compile_fragments(
+        &mut self,
+        source_name: &str,
+        source_contents: &str,
+    ) -> Result<Increment, Errors> {
+        let (core, unit) = self.store.get_open_mut();
+
+        let mut increment = self
+            .frontend
+            .compile_fragments(unit, source_name, source_contents)
+            .map_err(into_errors)?;
+
+        let pass_errors = self.passes.run_default_passes(
+            &mut increment.hir,
+            &mut unit.assigner,
+            core,
+            PackageType::Lib,
+        );
+
+        if !pass_errors.is_empty() {
+            return Err(into_errors_with_source(pass_errors, &unit.sources));
+        }
+
+        Ok(increment)
+    }
+
+    /// Compiles an entry expression.
+    ///
+    /// This method returns the AST and HIR packages that were created as a result of
+    /// the compilation, however it does *not* update the current compilation.
+    ///
+    /// The caller can use the returned packages to perform passes,
+    /// get information about the newly added items, or do other modifications.
+    /// It is then the caller's responsibility to merge
+    /// these packages into the current `CompileUnit` using the `update()` method.
+    pub fn compile_expr(&mut self, expr: &str) -> Result<Increment, Errors> {
+        let (core, unit) = self.store.get_open_mut();
+
+        let mut increment = self
+            .frontend
+            .compile_expr(unit, "<entry>", expr)
+            .map_err(into_errors)?;
+
+        let pass_errors = self.passes.run_default_passes(
+            &mut increment.hir,
+            &mut unit.assigner,
+            core,
+            PackageType::Lib,
+        );
+
+        if !pass_errors.is_empty() {
+            return Err(into_errors_with_source(pass_errors, &unit.sources));
+        }
+
+        Ok(increment)
+    }
+
+    /// Updates the current compilation with the AST and HIR packages,
+    /// and any associated context, returned from a previous incremental compilation.
+    pub fn update(&mut self, new: Increment) {
+        let (_, unit) = self.store.get_open_mut();
+
+        self.frontend.update(unit, new);
+    }
+
+    /// Returns a reference to the underlying package store.
+    #[must_use]
+    pub fn package_store(&self) -> &PackageStore {
+        self.store.package_store()
+    }
+
+    /// Returns ID of the current `CompileUnit`.
+    #[must_use]
+    pub fn package_id(&self) -> PackageId {
+        self.store.open_package_id()
+    }
+
+    /// Returns the ID of the source package created from the sources
+    /// passed in during inital creation.
+    #[must_use]
+    pub fn source_package_id(&self) -> PackageId {
+        self.source_package_id
+    }
+
+    /// Consumes the incremental compiler and returns an immutable package store.
+    /// This method can be used to finalize the compilation.
+    #[must_use]
+    pub fn into_package_store(self) -> (PackageStore, PackageId) {
+        self.store.into_package_store()
+    }
+}
+
+fn into_errors_with_source<T>(errors: Vec<T>, sources: &SourceMap) -> Errors
+where
+    compile::Error: From<T>,
+{
+    errors
+        .into_iter()
+        .map(|e| WithSource::from_map(sources, e.into()))
+        .collect()
+}
+
+fn into_errors<T>(errors: Vec<WithSource<T>>) -> Errors
+where
+    compile::Error: From<T>,
+    T: Diagnostic,
+{
+    errors
+        .into_iter()
+        .map(qsc_frontend::error::WithSource::into_with_source)
+        .collect()
+}

compiler/qsc/src/interpret/debug/tests.rs

@@ -87,7 +87,7 @@ fn stack_traces_can_cross_eval_session_and_file_boundaries() {
                              at Adjoint Test.C in 1.qs
                              at Adjoint Test.B in 1.qs
                              at Adjoint Test2.A in 2.qs
-                             at Z in <expression>
+                             at Z in line_0
                     "#};
             assert_eq!(expectation, stack_trace);
         }