From 33fa9530f165a5e5e1d367a5b51b291c2bb15d1f Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Wed, 7 Aug 2024 20:54:52 -0700 Subject: [PATCH] Support `module.sc` files in subfolders (#3213) This PR implements support for per-subfolder `build.sc` files, named `module.sc`. This allows large builds to be split up into multiple `build.sc` files each relevant to the sub-folder that they are placed in, rather than having a multi-thousand-line `build.sc` in the root configuring Mill for the entire codebase. ## Semantics 1. The `build.sc` in the project root and all nested `module.sc` files are recursively within the Mill directory tree are discovered (TODO ignoring files listed in `.millignore`), and compiled together 2. We ignore any subfolders that have their own `build.sc` file, indicating that they are the root of their own project and not part of the enclosing folder. 3. Each `foo/bar/qux/build.sc` file is compiled into a `millbuild.foo.bar.qux` `package object`, with the `build.sc` and `module.sc` files being compiled into a `millbuild` `package object` (rather than a plain `object` in the status quo) 4. An `object blah extends Module` within each `foo/bar/qux/build.sc` file can be referenced in code via `foo.bar.qux.blah`, or referenced from the command line via `foo.bar.qux.blah` 5. The base modules of `module.sc` files do not have the `MainModule` tasks: `init`, `version`, `clean`, etc.. Only the base module of the root `build.sc` file has those ## Design ### Uniform vs Non-uniform hierarchy One design decision here is whether a `module.sc` file in a subfolder `foo/bar/` containing `object qux{ def baz }` would have their targets referenced via `foo.bar.qux.baz` syntax, or via some alternative e.g. `foo/bar/qux.baz`. A non-uniform hierarchy `foo/bar/qux.baz` would be similar to how Bazel treats folders v.s. targets non-uniformly `foo/bar:qux-baz`, and also similar to how external modules in Mill are handled e.g. `mill.idea.GenIdea/idea`, as well as existing foreign modules. However, it introduces significant user-facing complexity: 1. What's the difference between `foo/bar/qux.baz` vs `foo/bar.qux.baz` or `foo/bar/qux/baz`? 2. What query syntax would we use to query targets in all nested `module.sc` files rather than just the top-level one e.g. `__.compile`? 3. Would there be a correspondingly different way of referencing nested `module.sc` modules and targets in Scala code as well? Bazel has significant complexity to handle these cases, e.g. query via `...` vs `:all` vs `*`. It works, but it does complicate the user-facing semantics. The alternative of a uniform hierarchy also has downsides: 1. How do you go from a module name e.g. `foo.bar.qux.baz` to the `build.sc` or `module.sc` file in which it is defined? 2. If a module is defined in both the root `build.sc` and in a nested `module.sc`, what happens? I decided to go with a uniform hierarchy where everything, both in top-level `build.sc` and in nested `module.sc`, end up squashed together in a single uniform `foo.bar.qux.baz` hierarchy. ### Package Objects The goal of this is to try and make modules defined in nested `module.sc` files "look the same" as modules defined in the root `build.sc`. There are two possible approaches: 1. Splice the source code of the various nested `module.sc` files into the top-level `object build`. This is possible, but very complex and error prone. Especially when it comes to reporting proper error locations in stack traces (filename/linenumber), this will likely require a custom compiler plugin similar to the `LineNumberPlugin` we have today 5. Convert the `object`s into `package object`s, such that module tree defined in the root `build.sc` becomes synonymous with the JVM package tree. While the `package object`s will cause the compiler to synthesize `object package { ... }` wrappers, that is mostly hidden from the end user. I decided to go with (2) because it seemed much simpler, making use of existing language features rather than trying to force the behavior we want using compiler hackery. Although `package object`s may go away at some point in Scala 3, they should be straightforward to replace with explicit `export foo.*` statements when that time comes. ### Existing Foreign Modules Mill already supports existing `foo.sc` files which support targets and modules being defined within them, but does not support referencing them from the command line. I have removed the ability to define targets and modules in random `foo.sc` files. We should encourage people to put things in `module.sc`, since that would allow the user to co-locate the build logic within the folder containing the files it is related to, rather than as a bunch of loose `foo.sc` scripts. Removing support for modules/targets in `foo.sc` files greatly simplifies the desugaring of these scripts, and since we are already making a breaking change by overhauling how per-folder `module.sc` files work we might as well bundle this additional breakage together (rather than making multiple breaking changes in series) ### `build.sc`/`module.sc` file discovery For this implementation, I chose to make `module.sc` files discovered automatically by traversing the filesystem: we recursively walk the subfolders of the root `build.sc` project, look for any files named `module.sc`. We only traverse folders with `module.sc` files to avoid having to traverse the entire filesystem structure every time. Empty `module.sc` files can be used as necessary to allow `module.sc` files to be placed deeper in the folder tree This matches the behavior of Bazel and SBT in discovering their `BUILD`/`build.sbt` files, and notably goes against Maven/Gradle which require submodules/subprojects to be declared in the top level build config. This design has the following characteristics: 1. In future, if we wish to allow `mill` invocations from within a subfolder, the distinction between `build.sc` and `module.sc` allows us to easily find the "enclosing" project root. 2. It ensures that any folders containing `build.sc`/`module.sc` files that accidentally get included within a Mill build do not end up getting picked up and confusing the top-level build, because we automatically skip any subfolders containing `build.sc` 3. Similarly, it ensures that adding a `build.sc` file "enclosing" an existing project, it would not affect Mill invocations in the inner project, because we only walk to the nearest enclosing `build.sc` file to find the project root 4. We do not automatically traverse deeply into sub-folders to discover `module.sc` files, which means that it should be almost impossible to accidentally pick up `module.sc` files that happen to be on the filesystem but you did not intend to include in the build This mechanism should do the right thing 99.9% of the time. For the last 0.1% where it doesn't do the right thing, we can add a `.millignore`/`.config/millignore` file to support ignoring things we don't want picked up, but I expect that should be a very rare edge case ## Task Resolution I have aimed to keep the logic in `resolve/` mostly intact. The core change is replacing `rootModule: BaseModule` with `baseModules: BaseModuleTree`, which provides enough metadata to allow `resolveDirectChildren` and `resolveTransitiveChildren` to find `BaseModule`s in sub-folders in addition to `Module` `object`s nested within the parent `Module`. Other than that, the logic should be basically unchanged, which hopefully should mitigate the risk of introducing new bugs ## Compatibility This change is not binary compatible, and the change in the `.sc` file desugaring is invasive enough we should consider it a major breaking change. This will need to go into Mill 0.12.x ## Out-Of-Scope/TODO 1. Running `mill` without a subfolder of the enclosing project. Shouldn't be hard to add given this design, but the PR is complicated enough as is that I'd like to leave it for follow up 2. Error reporting when a module is duplicated in an enclosing `object` and in a nested `module.sc` file. Again, probably not hard to add, but can happen in a follow up Pull request: https://github.com/com-lihaoyi/mill/pull/3213 --- bsp/worker/src/mill/bsp/worker/State.scala | 4 +- build.sc | 13 +- ci/mill-bootstrap.patch | 13 ++ .../src/mill/contrib/bloop/BloopImpl.scala | 2 +- .../contrib/scoverage/ScoverageReport.scala | 4 +- example/javaweb/4-hello-micronaut/build.sc | 2 +- .../src/main/resources/application.properties | 2 +- example/misc/8-multi-build-file/bar/module.sc | 0 .../misc/8-multi-build-file/bar/qux/module.sc | 5 + .../bar/qux/module/src/BarQux.scala | 9 + example/misc/8-multi-build-file/build.sc | 30 +++ example/misc/8-multi-build-file/foo/module.sc | 6 + .../misc/8-multi-build-file/foo/src/Foo.scala | 14 ++ .../mill/integration/ExampleTestSuite.scala | 13 +- idea/src/mill/idea/GenIdeaImpl.scala | 8 +- .../test/src/CompileErrorTests.scala | 2 +- .../test/src/CrossCollisionsTests.scala | 3 +- .../src/MultipleTopLevelModulesTests.scala | 2 +- .../test/src/DocAnnotationsTests.scala | 1 - .../feature/foreign/repo/conflict/build.sc | 23 --- .../foreign/repo/conflict/inner/build.sc | 3 - .../feature/foreign/repo/outer/build.sc | 30 --- .../feature/foreign/repo/outer/inner/build.sc | 13 -- .../feature/foreign/repo/project/build.sc | 114 ----------- .../foreign/repo/project/inner/build.sc | 13 -- .../feature/foreign/repo/project/other.sc | 13 -- .../foreign/test/src/ForeignBuildsTest.scala | 25 --- .../test/src/ForeignConflictTest.scala | 23 --- integration/ide/bsp-modules/repo/build.sc | 11 +- .../ide/bsp-modules/repo/proj1/build.sc | 2 +- .../ide/bsp-modules/repo/proj1/module.sc | 6 + .../ide/bsp-modules/repo/proj2/build.sc | 2 +- .../ide/bsp-modules/repo/proj2/module.sc | 6 + .../ide/bsp-modules/repo/proj3/build.sc | 2 +- .../ide/bsp-modules/repo/proj3/module.sc | 6 + .../test/src/BspModulesTests.scala | 4 +- .../invalidation-foreign/repo/build.sc | 5 - .../repo/foreignA/build.sc | 13 -- .../repo/foreignB/build.sc | 5 - .../src/ScriptsInvalidationForeignTests.scala | 66 ------- .../repo/-#!+\342\206\222&%=~/module.sc" | 0 .../repo/{-#+&%.sc => -#+&%/module.sc} | 0 .../repo/a/{inputA.sc => module.sc} | 0 .../repo/b/{inputB.sc => module.sc} | 0 .../invalidation/invalidation/repo/build.sc | 25 +-- .../repo/{inputC.sc => c/module.sc} | 0 .../repo/e/{inputE.sc => module.sc} | 3 +- .../integration/IntegrationTestSuite.scala | 5 +- main/define/src/mill/define/BaseModule.scala | 42 +++- .../src/mill/define/BaseModuleTree.scala | 41 ++++ main/eval/src/mill/eval/Evaluator.scala | 6 +- main/eval/src/mill/eval/EvaluatorImpl.scala | 2 +- main/eval/src/mill/eval/GroupEvaluator.scala | 2 +- main/eval/src/mill/eval/Plan.scala | 2 +- main/resolve/src/mill/resolve/Resolve.scala | 78 +++++--- .../src/mill/resolve/ResolveCore.scala | 181 ++++++++++++------ .../test/src/mill/main/ResolveTests.scala | 4 +- main/src/mill/main/MainModule.scala | 66 ++----- main/src/mill/main/RootModule.scala | 58 ++++-- main/src/mill/main/RunScript.scala | 2 +- main/src/mill/main/TokenReaders.scala | 2 +- .../src/mill/testkit/MillTestkit.scala | 4 +- runner/src/mill/runner/FileImportGraph.scala | 38 ++-- .../src/mill/runner/MillBuildBootstrap.scala | 84 +++++--- .../src/mill/runner/MillBuildRootModule.scala | 98 +++++----- runner/src/mill/runner/RunnerState.scala | 5 +- scalalib/src/mill/scalalib/Dependency.scala | 4 +- scalalib/src/mill/scalalib/GenIdeaImpl.scala | 2 +- 68 files changed, 616 insertions(+), 656 deletions(-) create mode 100644 example/misc/8-multi-build-file/bar/module.sc create mode 100644 example/misc/8-multi-build-file/bar/qux/module.sc create mode 100644 example/misc/8-multi-build-file/bar/qux/module/src/BarQux.scala create mode 100644 example/misc/8-multi-build-file/build.sc create mode 100644 example/misc/8-multi-build-file/foo/module.sc create mode 100644 example/misc/8-multi-build-file/foo/src/Foo.scala delete mode 100644 integration/feature/foreign/repo/conflict/build.sc delete mode 100644 integration/feature/foreign/repo/conflict/inner/build.sc delete mode 100644 integration/feature/foreign/repo/outer/build.sc delete mode 100644 integration/feature/foreign/repo/outer/inner/build.sc delete mode 100644 integration/feature/foreign/repo/project/build.sc delete mode 100644 integration/feature/foreign/repo/project/inner/build.sc delete mode 100644 integration/feature/foreign/repo/project/other.sc delete mode 100644 integration/feature/foreign/test/src/ForeignBuildsTest.scala delete mode 100644 integration/feature/foreign/test/src/ForeignConflictTest.scala create mode 100644 integration/ide/bsp-modules/repo/proj1/module.sc create mode 100644 integration/ide/bsp-modules/repo/proj2/module.sc create mode 100644 integration/ide/bsp-modules/repo/proj3/module.sc delete mode 100644 integration/invalidation/invalidation-foreign/repo/build.sc delete mode 100644 integration/invalidation/invalidation-foreign/repo/foreignA/build.sc delete mode 100644 integration/invalidation/invalidation-foreign/repo/foreignB/build.sc delete mode 100644 integration/invalidation/invalidation-foreign/test/src/ScriptsInvalidationForeignTests.scala rename "integration/invalidation/invalidation/repo/-#!+\342\206\222&%=~/inputSymbols.sc" => "integration/invalidation/invalidation/repo/-#!+\342\206\222&%=~/module.sc" (100%) rename integration/invalidation/invalidation/repo/{-#+&%.sc => -#+&%/module.sc} (100%) rename integration/invalidation/invalidation/repo/a/{inputA.sc => module.sc} (100%) rename integration/invalidation/invalidation/repo/b/{inputB.sc => module.sc} (100%) rename integration/invalidation/invalidation/repo/{inputC.sc => c/module.sc} (100%) rename integration/invalidation/invalidation/repo/e/{inputE.sc => module.sc} (53%) create mode 100644 main/define/src/mill/define/BaseModuleTree.scala diff --git a/bsp/worker/src/mill/bsp/worker/State.scala b/bsp/worker/src/mill/bsp/worker/State.scala index f9a97a75840..d5efe434481 100644 --- a/bsp/worker/src/mill/bsp/worker/State.scala +++ b/bsp/worker/src/mill/bsp/worker/State.scala @@ -9,7 +9,7 @@ import mill.eval.Evaluator private class State(evaluators: Seq[Evaluator], debug: String => Unit) { lazy val bspModulesById: Map[BuildTargetIdentifier, (BspModule, Evaluator)] = { val modules: Seq[(Module, Seq[Module], Evaluator)] = evaluators - .map(ev => (ev.rootModule, JavaModuleUtils.transitiveModules(ev.rootModule), ev)) + .flatMap(ev => ev.rootModules.map(rm => (rm, JavaModuleUtils.transitiveModules(rm), ev))) val map = modules .flatMap { case (rootModule, otherModules, eval) => @@ -30,7 +30,7 @@ private class State(evaluators: Seq[Evaluator], debug: String => Unit) { map } - lazy val rootModules: Seq[mill.define.BaseModule] = evaluators.map(_.rootModule) + lazy val rootModules: Seq[mill.define.BaseModule] = evaluators.flatMap(_.rootModules) lazy val bspIdByModule: Map[BspModule, BuildTargetIdentifier] = bspModulesById.view.mapValues(_._1).map(_.swap).toMap diff --git a/build.sc b/build.sc index 09fca0f4e59..1a81e907dc4 100644 --- a/build.sc +++ b/build.sc @@ -514,7 +514,18 @@ trait MillStableScalaModule extends MillPublishScalaModule with Mima { ), ProblemFilter.exclude[ReversedMissingMethodProblem]( "mill.scalalib.JavaModule.mill$scalalib$JavaModule$$super$runMain" - ) + ), + + // Not sure why mima is picking up this stuff which is private[mill] + ProblemFilter.exclude[Problem]("mill.resolve.*.resolve0"), + ProblemFilter.exclude[Problem]("mill.resolve.*.resolveRootModule"), + + // These methods are private so it doesn't matter + ProblemFilter.exclude[ReversedMissingMethodProblem]("mill.resolve.Resolve.handleResolved"), + ProblemFilter.exclude[Problem]("mill.resolve.*.resolveNonEmptyAndHandle*"), + ProblemFilter.exclude[Problem]("mill.resolve.ResolveCore*"), + ProblemFilter.exclude[InheritedNewAbstractMethodProblem]("mill.main.MainModule.mill$define$BaseModule0$_setter_$watchedValues_="), + ProblemFilter.exclude[InheritedNewAbstractMethodProblem]("mill.main.MainModule.mill$define$BaseModule0$_setter_$evalWatchedValues_="), ) def mimaPreviousVersions: T[Seq[String]] = Settings.mimaBaseVersions diff --git a/ci/mill-bootstrap.patch b/ci/mill-bootstrap.patch index e69de29bb2d..8276fc52892 100644 --- a/ci/mill-bootstrap.patch +++ b/ci/mill-bootstrap.patch @@ -0,0 +1,13 @@ +diff --git a/build.sc b/build.sc +index 526ebaa9cf..616be491f9 100644 +--- a/build.sc ++++ b/build.sc +@@ -1968,7 +1968,7 @@ def uploadToGithub(authKey: String) = T.command { + + private def resolveTasks[T](taskNames: String*): Seq[NamedTask[T]] = { + mill.resolve.Resolve.Tasks.resolve( +- build, ++ build.`package`, + taskNames, + SelectMode.Separated + ).map(x => x.asInstanceOf[Seq[mill.define.NamedTask[T]]]).getOrElse(???) \ No newline at end of file diff --git a/contrib/bloop/src/mill/contrib/bloop/BloopImpl.scala b/contrib/bloop/src/mill/contrib/bloop/BloopImpl.scala index 3d27d323cfd..88bd29fbddc 100644 --- a/contrib/bloop/src/mill/contrib/bloop/BloopImpl.scala +++ b/contrib/bloop/src/mill/contrib/bloop/BloopImpl.scala @@ -120,7 +120,7 @@ class BloopImpl(evs: () => Seq[Evaluator], wd: os.Path) extends ExternalModule { val evals = evs() evals.flatMap { eval => if (eval != null) - JavaModuleUtils.transitiveModules(eval.rootModule, accept) + eval.rootModules.flatMap(JavaModuleUtils.transitiveModules(_, accept)) .collect { case jm: JavaModule => jm } else Seq.empty diff --git a/contrib/scoverage/src/mill/contrib/scoverage/ScoverageReport.scala b/contrib/scoverage/src/mill/contrib/scoverage/ScoverageReport.scala index a89de439996..5dd757fcbbf 100644 --- a/contrib/scoverage/src/mill/contrib/scoverage/ScoverageReport.scala +++ b/contrib/scoverage/src/mill/contrib/scoverage/ScoverageReport.scala @@ -91,7 +91,7 @@ trait ScoverageReport extends Module { dataTargets: String ): Task[PathRef] = { val sourcesTasks: Seq[Task[Seq[PathRef]]] = Resolve.Tasks.resolve( - evaluator.rootModule, + evaluator.rootModules, Seq(sources), SelectMode.Separated ) match { @@ -99,7 +99,7 @@ trait ScoverageReport extends Module { case Right(tasks) => tasks.asInstanceOf[Seq[Task[Seq[PathRef]]]] } val dataTasks: Seq[Task[PathRef]] = Resolve.Tasks.resolve( - evaluator.rootModule, + evaluator.rootModules, Seq(dataTargets), SelectMode.Separated ) match { diff --git a/example/javaweb/4-hello-micronaut/build.sc b/example/javaweb/4-hello-micronaut/build.sc index 2ec70e65d4b..6d37e1a9688 100644 --- a/example/javaweb/4-hello-micronaut/build.sc +++ b/example/javaweb/4-hello-micronaut/build.sc @@ -70,7 +70,7 @@ trait MicronautModule extends MavenModule{ > mill runBackground -> curl http://localhost:8087/hello +> curl http://localhost:8088/hello ...Hello World... > mill clean runBackground diff --git a/example/javaweb/4-hello-micronaut/src/main/resources/application.properties b/example/javaweb/4-hello-micronaut/src/main/resources/application.properties index 07d288f675c..0f3cd29637a 100644 --- a/example/javaweb/4-hello-micronaut/src/main/resources/application.properties +++ b/example/javaweb/4-hello-micronaut/src/main/resources/application.properties @@ -1,3 +1,3 @@ #Tue Jun 18 12:49:41 UTC 2024 micronaut.application.name=default -micronaut.server.port=8087 +micronaut.server.port=8088 diff --git a/example/misc/8-multi-build-file/bar/module.sc b/example/misc/8-multi-build-file/bar/module.sc new file mode 100644 index 00000000000..e69de29bb2d diff --git a/example/misc/8-multi-build-file/bar/qux/module.sc b/example/misc/8-multi-build-file/bar/qux/module.sc new file mode 100644 index 00000000000..c28b0b95d30 --- /dev/null +++ b/example/misc/8-multi-build-file/bar/qux/module.sc @@ -0,0 +1,5 @@ +import mill._, scalalib._ + +object module extends build.MyModule { + def ivyDeps = Agg(ivy"com.lihaoyi::scalatags:0.8.2") +} diff --git a/example/misc/8-multi-build-file/bar/qux/module/src/BarQux.scala b/example/misc/8-multi-build-file/bar/qux/module/src/BarQux.scala new file mode 100644 index 00000000000..167606ea0a7 --- /dev/null +++ b/example/misc/8-multi-build-file/bar/qux/module/src/BarQux.scala @@ -0,0 +1,9 @@ +package bar.qux +import scalatags.Text.all._ +object BarQux { + def printText(text: String): Unit = { + val value = p("world") + println("BarQux.value: " + value) + } + def main(args: Array[String]) = printText(args(0)) +} diff --git a/example/misc/8-multi-build-file/build.sc b/example/misc/8-multi-build-file/build.sc new file mode 100644 index 00000000000..9221a52d6b2 --- /dev/null +++ b/example/misc/8-multi-build-file/build.sc @@ -0,0 +1,30 @@ +import mill._, scalalib._ + +trait MyModule extends ScalaModule { + def scalaVersion = "2.13.11" +} + +// Example Docs + + +/** Usage + +> ./mill resolve __ +bar +... +bar.qux.module +... +bar.qux.module.compile +... +foo +... +foo.compile + +> ./mill bar.qux.module.compile + +> ./mill foo.compile + +> ./mill foo.run --foo-text hello --bar-qux-text world +Foo.value: hello +BarQux.value:

world

+*/ diff --git a/example/misc/8-multi-build-file/foo/module.sc b/example/misc/8-multi-build-file/foo/module.sc new file mode 100644 index 00000000000..5803d1e9ae1 --- /dev/null +++ b/example/misc/8-multi-build-file/foo/module.sc @@ -0,0 +1,6 @@ +import mill._, scalalib._ + +object build extends RootModule with MyModule { + def moduleDeps = Seq(bar.qux.module) + def ivyDeps = Agg(ivy"com.lihaoyi::mainargs:0.4.0") +} diff --git a/example/misc/8-multi-build-file/foo/src/Foo.scala b/example/misc/8-multi-build-file/foo/src/Foo.scala new file mode 100644 index 00000000000..7e1c4d86e91 --- /dev/null +++ b/example/misc/8-multi-build-file/foo/src/Foo.scala @@ -0,0 +1,14 @@ +package foo +import mainargs.{main, ParserForMethods, arg} +object Foo { + val value = "hello" + + @main + def main(@arg(name = "foo-text") fooText: String, + @arg(name = "bar-qux-text") barQuxText: String): Unit = { + println("Foo.value: " + Foo.value) + bar.qux.BarQux.printText(barQuxText) + } + + def main(args: Array[String]): Unit = ParserForMethods(this).runOrExit(args) +} diff --git a/example/src/mill/integration/ExampleTestSuite.scala b/example/src/mill/integration/ExampleTestSuite.scala index d0f7e3109a6..b2dc1567c60 100644 --- a/example/src/mill/integration/ExampleTestSuite.scala +++ b/example/src/mill/integration/ExampleTestSuite.scala @@ -71,18 +71,17 @@ object ExampleTestSuite extends IntegrationTestSuite { } test("exampleUsage") { + val parsed = upickle.default.read[Seq[(String, String)]](sys.env("MILL_EXAMPLE_PARSED")) val usageComment = parsed.collect { case ("example", txt) => txt }.mkString("\n\n") val commandBlocks = ("\n" + usageComment.trim).split("\n> ").filter(_.nonEmpty) retryOnTimeout(3) { - try { - for (commandBlock <- commandBlocks) processCommandBlock(workspaceRoot, commandBlock) - if (integrationTestMode != "fork") evalStdout("shutdown") - } finally { - try os.remove.all(workspaceRoot / "out") - catch { case e: Throwable => /*do nothing*/ } - } + try os.remove.all(workspaceRoot / "out") + catch { case e: Throwable => /*do nothing*/ } + + for (commandBlock <- commandBlocks) processCommandBlock(workspaceRoot, commandBlock) + if (integrationTestMode != "fork") evalStdout("shutdown") } } } diff --git a/idea/src/mill/idea/GenIdeaImpl.scala b/idea/src/mill/idea/GenIdeaImpl.scala index 9c84ce5a795..6e68c51ab44 100755 --- a/idea/src/mill/idea/GenIdeaImpl.scala +++ b/idea/src/mill/idea/GenIdeaImpl.scala @@ -22,7 +22,7 @@ case class GenIdeaImpl( )(implicit ctx: Ctx) { import GenIdeaImpl._ - val workDir: os.Path = evaluators.head.rootModule.millSourcePath + val workDir: os.Path = evaluators.head.rootModules.head.millSourcePath val ideaDir: os.Path = workDir / ".idea" val ideaConfigVersion = 4 @@ -67,7 +67,9 @@ case class GenIdeaImpl( fetchMillModules: Boolean = true ): Seq[(os.SubPath, scala.xml.Node)] = { - val rootModules = evaluators.zipWithIndex.map { case (ev, idx) => (ev.rootModule, ev, idx) } + val rootModules = evaluators.zipWithIndex.map { case (ev, idx) => + (ev.rootModules.head, ev, idx) + } val transitive: Seq[(BaseModule, Seq[Module], Evaluator, Int)] = rootModules .map { case (rootModule, ev, idx) => (rootModule, JavaModuleUtils.transitiveModules(rootModule), ev, idx) @@ -118,7 +120,7 @@ case class GenIdeaImpl( } // is head the right one? - val buildDepsPaths = Classpath.allJars(evaluators.head.rootModule.getClass.getClassLoader) + val buildDepsPaths = Classpath.allJars(evaluators.head.rootModules.head.getClass.getClassLoader) .map(url => os.Path(java.nio.file.Paths.get(url.toURI))) def resolveTasks: Map[Evaluator, Seq[Task[ResolvedModule]]] = diff --git a/integration/failure/compile-error/test/src/CompileErrorTests.scala b/integration/failure/compile-error/test/src/CompileErrorTests.scala index 42001c73c84..50ea4e6181a 100644 --- a/integration/failure/compile-error/test/src/CompileErrorTests.scala +++ b/integration/failure/compile-error/test/src/CompileErrorTests.scala @@ -15,7 +15,7 @@ object CompileErrorTests extends IntegrationTestSuite { assert(res.err.contains("""println(doesntExist)""")) assert(res.err.contains("""qux.sc:3:34: type mismatch;""")) assert(res.err.contains( - """build.sc:8:5: value noSuchMethod is not a member of object build.this.foo""" + """build.sc:8:5: value noSuchMethod is not a member of object""" )) } } diff --git a/integration/failure/cross-collisions/test/src/CrossCollisionsTests.scala b/integration/failure/cross-collisions/test/src/CrossCollisionsTests.scala index b722d4902b3..11f35cef881 100644 --- a/integration/failure/cross-collisions/test/src/CrossCollisionsTests.scala +++ b/integration/failure/cross-collisions/test/src/CrossCollisionsTests.scala @@ -9,9 +9,10 @@ object CrossCollisionsTests extends IntegrationTestSuite { test("detect-collision") { val res = evalStdout("resolve", "foo._") assert(!res.isSuccess) + assert(res.err.contains("Cross module ")) assert( res.err.contains( - "Cross module millbuild.build#foo contains colliding cross values: List(List(a, b), List(c)) and List(List(a), List(b, c))" + " contains colliding cross values: List(List(a, b), List(c)) and List(List(a), List(b, c))" ) ) } diff --git a/integration/failure/multiple-top-level-modules/test/src/MultipleTopLevelModulesTests.scala b/integration/failure/multiple-top-level-modules/test/src/MultipleTopLevelModulesTests.scala index 90f61a99812..2254a631414 100644 --- a/integration/failure/multiple-top-level-modules/test/src/MultipleTopLevelModulesTests.scala +++ b/integration/failure/multiple-top-level-modules/test/src/MultipleTopLevelModulesTests.scala @@ -10,7 +10,7 @@ object MultipleTopLevelModulesTests extends IntegrationTestSuite { val res = evalStdout("resolve", "_") assert(!res.isSuccess) assert(res.err.contains( - "Only one RootModule can be defined in a build, not 2: millbuild.build$bar$,millbuild.build$foo$" + "Only one RootModule can be defined in a build, not 2:" )) } } diff --git a/integration/feature/docannotations/test/src/DocAnnotationsTests.scala b/integration/feature/docannotations/test/src/DocAnnotationsTests.scala index 07428e428e7..2277900de30 100644 --- a/integration/feature/docannotations/test/src/DocAnnotationsTests.scala +++ b/integration/feature/docannotations/test/src/DocAnnotationsTests.scala @@ -37,7 +37,6 @@ object DocAnnotationsTests extends IntegrationTestSuite { assert(eval("inspect", "core.target")) val target = ujson.read(meta("inspect"))("value").str - pprint.log(target) assert( globMatches( """core.target(build.sc:...) diff --git a/integration/feature/foreign/repo/conflict/build.sc b/integration/feature/foreign/repo/conflict/build.sc deleted file mode 100644 index 1acddf2d8e5..00000000000 --- a/integration/feature/foreign/repo/conflict/build.sc +++ /dev/null @@ -1,23 +0,0 @@ -import $file.inner.{build => innerBuild} -import mill._ - -// In this build, we have a local module targeting -// the 'inner sub-directory, and an imported foreign -// module in that same directory. Their sourcePaths -// should be the same, but their dest paths should -// be different to avoid both modules over-writing -// each other's caches . - -def checkPaths: T[Unit] = T { - if (innerBuild.millSourcePath != inner.millSourcePath) - throw new Exception("Source paths should be the same") -} - -def checkDests: T[Unit] = T { - if (innerBuild.selfDest == inner.selfDest) - throw new Exception("Dest paths should be different") -} - -object inner extends mill.Module { - def selfDest = T { T.dest / os.up } -} diff --git a/integration/feature/foreign/repo/conflict/inner/build.sc b/integration/feature/foreign/repo/conflict/inner/build.sc deleted file mode 100644 index 7c1414a1213..00000000000 --- a/integration/feature/foreign/repo/conflict/inner/build.sc +++ /dev/null @@ -1,3 +0,0 @@ -import mill._ - -def selfDest = T { T.dest / os.up } diff --git a/integration/feature/foreign/repo/outer/build.sc b/integration/feature/foreign/repo/outer/build.sc deleted file mode 100644 index 8166603e031..00000000000 --- a/integration/feature/foreign/repo/outer/build.sc +++ /dev/null @@ -1,30 +0,0 @@ -import $file.inner.build -import mill._ - -trait PathAware extends mill.Module { - def selfPath = T { millSourcePath } -} - -trait DestAware extends mill.Module { - def selfDest = T { T.dest / os.up } -} - -object sub extends PathAware with DestAware { - object sub extends PathAware with DestAware -} - -object sourcepathmod extends mill.Module { - def selfDest = T { T.dest / os.up } - - object jvm extends mill.Module { - def selfDest = T { T.dest / os.up } - def millSourcePath = sourcepathmod.millSourcePath - def sources = T.sources(millSourcePath / "src", millSourcePath / "src-jvm") - } - - object js extends mill.Module { - def selfDest = T { T.dest / os.up } - def millSourcePath = sourcepathmod.millSourcePath - def sources = T.sources(millSourcePath / "src", millSourcePath / "src-js") - } -} diff --git a/integration/feature/foreign/repo/outer/inner/build.sc b/integration/feature/foreign/repo/outer/inner/build.sc deleted file mode 100644 index dd03d6938fa..00000000000 --- a/integration/feature/foreign/repo/outer/inner/build.sc +++ /dev/null @@ -1,13 +0,0 @@ -import mill._ - -trait PathAware extends mill.Module { - def selfPath = T { millSourcePath } -} - -trait DestAware extends mill.Module { - def selfDest = T { T.dest / os.up } -} - -object sub extends PathAware with DestAware { - object sub extends PathAware with DestAware -} diff --git a/integration/feature/foreign/repo/project/build.sc b/integration/feature/foreign/repo/project/build.sc deleted file mode 100644 index e71878ce9f5..00000000000 --- a/integration/feature/foreign/repo/project/build.sc +++ /dev/null @@ -1,114 +0,0 @@ -import $file.^.outer.build -import $file.inner.build -import $file.other - -import mill._ - -def assertPathsEqual(p1: os.Path, p2: os.Path): Unit = if (p1 != p2) throw new Exception( - s"Paths were not equal : \n- $p1 \n- $p2" -) -def assertPathsNotEqual(p1: os.Path, p2: os.Path): Unit = if (p1 == p2) throw new Exception( - s"Paths were equal : \n- $p1 \n- $p2" -) - -object sub extends PathAware with DestAware { - - object sub extends PathAware with DestAware - - object sub2 extends ^.outer.build.PathAware with ^.outer.build.DestAware - -} - -def checkProjectPaths = T { - val thisPath: os.Path = millSourcePath - assert(thisPath.last == "project") - assertPathsEqual(sub.selfPath(), thisPath / "sub") - assertPathsEqual(sub.sub.selfPath(), thisPath / "sub" / "sub") - assertPathsEqual(sub.sub2.selfPath(), thisPath / "sub" / "sub2") -} - -def checkInnerPaths = T { - val thisPath: os.Path = millSourcePath - assertPathsEqual(inner.build.millSourcePath, thisPath / "inner") - assertPathsEqual(inner.build.sub.selfPath(), thisPath / "inner" / "sub") - assertPathsEqual(inner.build.sub.sub.selfPath(), thisPath / "inner" / "sub" / "sub") -} - -def checkOuterPaths = T { - val thisPath: os.Path = millSourcePath - assertPathsEqual(^.outer.build.millSourcePath, thisPath / os.up / "outer") - assertPathsEqual(^.outer.build.sub.selfPath(), thisPath / os.up / "outer" / "sub") - assertPathsEqual(^.outer.build.sub.sub.selfPath(), thisPath / os.up / "outer" / "sub" / "sub") - - // covers the case where millSourcePath is modified in a submodule - assertPathsNotEqual( - ^.outer.build.sourcepathmod.jvm.selfDest(), - ^.outer.build.sourcepathmod.js.selfDest() - ) -} - -def checkOuterInnerPaths = T { - val thisPath: os.Path = millSourcePath - assertPathsEqual(^.outer.inner.build.millSourcePath, thisPath / os.up / "outer" / "inner") - assertPathsEqual(^.outer.inner.build.sub.selfPath(), thisPath / os.up / "outer" / "inner" / "sub") - assertPathsEqual( - ^.outer.inner.build.sub.sub.selfPath(), - thisPath / os.up / "outer" / "inner" / "sub" / "sub" - ) -} - -def checkOtherPaths = T { - val thisPath: os.Path = millSourcePath - assertPathsEqual(other.millSourcePath, thisPath) - assertPathsEqual(other.sub.selfPath(), thisPath / "sub") - assertPathsEqual(other.sub.sub.selfPath(), thisPath / "sub" / "sub") -} - -def checkProjectDests = T { - val outPath: os.Path = millSourcePath / "out" - assertPathsEqual(sub.selfDest(), outPath / "sub") - assertPathsEqual(sub.sub.selfDest(), outPath / "sub" / "sub") - assertPathsEqual(sub.sub2.selfDest(), outPath / "sub" / "sub2") -} - -def checkInnerDests = T { - val foreignOut: os.Path = millSourcePath / "out" / "foreign-modules" - assertPathsEqual(inner.build.sub.selfDest(), foreignOut / "inner" / "build" / "sub") - assertPathsEqual(inner.build.sub.sub.selfDest(), foreignOut / "inner" / "build" / "sub" / "sub") -} - -def checkOuterDests = T { - val foreignOut: os.Path = millSourcePath / "out" / "foreign-modules" - assertPathsEqual(^.outer.build.sub.selfDest(), foreignOut / "up-1" / "outer" / "build" / "sub") - assertPathsEqual( - ^.outer.build.sub.sub.selfDest(), - foreignOut / "up-1" / "outer" / "build" / "sub" / "sub" - ) -} - -def checkOuterInnerDests = T { - val foreignOut: os.Path = millSourcePath / "out" / "foreign-modules" - assertPathsEqual( - ^.outer.inner.build.sub.selfDest(), - foreignOut / "up-1" / "outer" / "inner" / "build" / "sub" - ) - assertPathsEqual( - ^.outer.inner.build.sub.sub.selfDest(), - foreignOut / "up-1" / "outer" / "inner" / "build" / "sub" / "sub" - ) -} - -def checkOtherDests = T { - val foreignOut: os.Path = millSourcePath / "out" / "foreign-modules" - assertPathsEqual(other.sub.selfDest(), foreignOut / "other" / "sub") - assertPathsEqual(other.sub.sub.selfDest(), foreignOut / "other" / "sub" / "sub") -} - -trait PathAware extends mill.Module { - - def selfPath = T { millSourcePath } -} - -trait DestAware extends mill.Module { - def selfDest = T { T.dest / os.up } -} diff --git a/integration/feature/foreign/repo/project/inner/build.sc b/integration/feature/foreign/repo/project/inner/build.sc deleted file mode 100644 index dd03d6938fa..00000000000 --- a/integration/feature/foreign/repo/project/inner/build.sc +++ /dev/null @@ -1,13 +0,0 @@ -import mill._ - -trait PathAware extends mill.Module { - def selfPath = T { millSourcePath } -} - -trait DestAware extends mill.Module { - def selfDest = T { T.dest / os.up } -} - -object sub extends PathAware with DestAware { - object sub extends PathAware with DestAware -} diff --git a/integration/feature/foreign/repo/project/other.sc b/integration/feature/foreign/repo/project/other.sc deleted file mode 100644 index dd03d6938fa..00000000000 --- a/integration/feature/foreign/repo/project/other.sc +++ /dev/null @@ -1,13 +0,0 @@ -import mill._ - -trait PathAware extends mill.Module { - def selfPath = T { millSourcePath } -} - -trait DestAware extends mill.Module { - def selfDest = T { T.dest / os.up } -} - -object sub extends PathAware with DestAware { - object sub extends PathAware with DestAware -} diff --git a/integration/feature/foreign/test/src/ForeignBuildsTest.scala b/integration/feature/foreign/test/src/ForeignBuildsTest.scala deleted file mode 100644 index 763f1bca018..00000000000 --- a/integration/feature/foreign/test/src/ForeignBuildsTest.scala +++ /dev/null @@ -1,25 +0,0 @@ -package mill.integration -package local - -import utest._ -import utest.framework.TestPath -import os.SubPath - -object ForeignBuildsTest extends IntegrationTestSuite { - override def buildPath: SubPath = os.sub / "project" / "build.sc" - - val tests: Tests = Tests { - initWorkspace() - def checkTarget()(implicit testPath: TestPath): Unit = assert(eval(testPath.value.last)) - "checkProjectPaths" - checkTarget() - "checkInnerPaths" - checkTarget() - "checkOuterPaths" - checkTarget() - "checkOuterInnerPaths" - checkTarget() - "checkOtherPaths" - checkTarget() - "checkProjectDests" - checkTarget() - "checkInnerDests" - checkTarget() - "checkOuterDests" - checkTarget() - "checkOuterInnerDests" - checkTarget() - "checkOtherDests" - checkTarget() - } -} diff --git a/integration/feature/foreign/test/src/ForeignConflictTest.scala b/integration/feature/foreign/test/src/ForeignConflictTest.scala deleted file mode 100644 index 7faddb1f40c..00000000000 --- a/integration/feature/foreign/test/src/ForeignConflictTest.scala +++ /dev/null @@ -1,23 +0,0 @@ -package mill.integration -package local - -import utest._ -import os.SubPath - -object ForeignConflictTest extends IntegrationTestSuite { - - override def buildPath: SubPath = os.sub / "conflict" / "build.sc" - - val tests: Tests = Tests { - initWorkspace() - test("test") - { - // see https://github.com/lihaoyi/mill/issues/302 - if (!mill.util.Util.java9OrAbove) { - assert( - eval("checkPaths"), - eval("checkDests") - ) - } - } - } -} diff --git a/integration/ide/bsp-modules/repo/build.sc b/integration/ide/bsp-modules/repo/build.sc index ffd07ee3641..2b84fa8fd3b 100644 --- a/integration/ide/bsp-modules/repo/build.sc +++ b/integration/ide/bsp-modules/repo/build.sc @@ -1,9 +1,6 @@ import mill._ import mill.api.{PathRef} import mill.scalalib._ -import $file.proj1.{build => proj1} -import $file.proj2.{build => proj2} -import $file.proj3.{build => proj3} trait HelloBspModule extends ScalaModule { def scalaVersion = sys.props.getOrElse("TEST_SCALA_2_13_VERSION", ???) @@ -12,17 +9,17 @@ trait HelloBspModule extends ScalaModule { object HelloBsp extends HelloBspModule { // Explicitly depends on proj1 - def moduleDeps: Seq[JavaModule] = Seq(proj1.proj1) + def moduleDeps: Seq[JavaModule] = Seq(build.proj1.module) // Explicitly depends on proj2 - def compileModuleDeps: Seq[JavaModule] = Seq(proj2.proj2) + def compileModuleDeps: Seq[JavaModule] = Seq(build.proj2.module) // Implicitly depends on proj3 via a target override def unmanagedClasspath: T[Agg[PathRef]] = T { - Agg(proj3.proj3.jar()) + Agg(build.proj3.module.jar()) } } def validate() = T.command { - val transitiveModules = mill.scalalib.internal.JavaModuleUtils.transitiveModules(build) + val transitiveModules = mill.scalalib.internal.JavaModuleUtils.transitiveModules(build.`package`) val file = T.dest / "transitive-modules.json" val moduleNames = transitiveModules.map(m => mill.scalalib.internal.ModuleUtils.moduleDisplayName(m) diff --git a/integration/ide/bsp-modules/repo/proj1/build.sc b/integration/ide/bsp-modules/repo/proj1/build.sc index 2e960d21c1f..54ab5c6355d 100644 --- a/integration/ide/bsp-modules/repo/proj1/build.sc +++ b/integration/ide/bsp-modules/repo/proj1/build.sc @@ -1,6 +1,6 @@ import mill._ import mill.scalalib._ -object proj1 extends ScalaModule { +object module extends ScalaModule { def scalaVersion = sys.props.getOrElse("TEST_SCALA_2_13_VERSION", ???) } diff --git a/integration/ide/bsp-modules/repo/proj1/module.sc b/integration/ide/bsp-modules/repo/proj1/module.sc new file mode 100644 index 00000000000..54ab5c6355d --- /dev/null +++ b/integration/ide/bsp-modules/repo/proj1/module.sc @@ -0,0 +1,6 @@ +import mill._ +import mill.scalalib._ + +object module extends ScalaModule { + def scalaVersion = sys.props.getOrElse("TEST_SCALA_2_13_VERSION", ???) +} diff --git a/integration/ide/bsp-modules/repo/proj2/build.sc b/integration/ide/bsp-modules/repo/proj2/build.sc index 8182930b31f..54ab5c6355d 100644 --- a/integration/ide/bsp-modules/repo/proj2/build.sc +++ b/integration/ide/bsp-modules/repo/proj2/build.sc @@ -1,6 +1,6 @@ import mill._ import mill.scalalib._ -object proj2 extends ScalaModule { +object module extends ScalaModule { def scalaVersion = sys.props.getOrElse("TEST_SCALA_2_13_VERSION", ???) } diff --git a/integration/ide/bsp-modules/repo/proj2/module.sc b/integration/ide/bsp-modules/repo/proj2/module.sc new file mode 100644 index 00000000000..54ab5c6355d --- /dev/null +++ b/integration/ide/bsp-modules/repo/proj2/module.sc @@ -0,0 +1,6 @@ +import mill._ +import mill.scalalib._ + +object module extends ScalaModule { + def scalaVersion = sys.props.getOrElse("TEST_SCALA_2_13_VERSION", ???) +} diff --git a/integration/ide/bsp-modules/repo/proj3/build.sc b/integration/ide/bsp-modules/repo/proj3/build.sc index 87a1708e405..54ab5c6355d 100644 --- a/integration/ide/bsp-modules/repo/proj3/build.sc +++ b/integration/ide/bsp-modules/repo/proj3/build.sc @@ -1,6 +1,6 @@ import mill._ import mill.scalalib._ -object proj3 extends ScalaModule { +object module extends ScalaModule { def scalaVersion = sys.props.getOrElse("TEST_SCALA_2_13_VERSION", ???) } diff --git a/integration/ide/bsp-modules/repo/proj3/module.sc b/integration/ide/bsp-modules/repo/proj3/module.sc new file mode 100644 index 00000000000..54ab5c6355d --- /dev/null +++ b/integration/ide/bsp-modules/repo/proj3/module.sc @@ -0,0 +1,6 @@ +import mill._ +import mill.scalalib._ + +object module extends ScalaModule { + def scalaVersion = sys.props.getOrElse("TEST_SCALA_2_13_VERSION", ???) +} diff --git a/integration/ide/bsp-modules/test/src/BspModulesTests.scala b/integration/ide/bsp-modules/test/src/BspModulesTests.scala index eaf9154278c..373dd8e4cf0 100644 --- a/integration/ide/bsp-modules/test/src/BspModulesTests.scala +++ b/integration/ide/bsp-modules/test/src/BspModulesTests.scala @@ -23,8 +23,8 @@ object BspModulesTests extends IntegrationTestSuite { "", // the root module has no segments at all "HelloBsp", "HelloBsp.test", - "foreign-modules.proj1.build.proj1", - "foreign-modules.proj2.build.proj2" + "proj1.module", + "proj2.module" // "foreign-modules.proj3.proj3" // still not detected ).sorted assert(readModules == expectedModules) diff --git a/integration/invalidation/invalidation-foreign/repo/build.sc b/integration/invalidation/invalidation-foreign/repo/build.sc deleted file mode 100644 index 18f68d95231..00000000000 --- a/integration/invalidation/invalidation-foreign/repo/build.sc +++ /dev/null @@ -1,5 +0,0 @@ -import mill._ - -def taskC = T { - println("c") -} diff --git a/integration/invalidation/invalidation-foreign/repo/foreignA/build.sc b/integration/invalidation/invalidation-foreign/repo/foreignA/build.sc deleted file mode 100644 index 2272f83511f..00000000000 --- a/integration/invalidation/invalidation-foreign/repo/foreignA/build.sc +++ /dev/null @@ -1,13 +0,0 @@ -import mill._ -import $file.^.foreignB.build -import $file.^.build - -def taskA = T { - ^.foreignB.build.taskB() - println("a") -} - -def taskD = T { - ^.build.taskC() - println("d") -} diff --git a/integration/invalidation/invalidation-foreign/repo/foreignB/build.sc b/integration/invalidation/invalidation-foreign/repo/foreignB/build.sc deleted file mode 100644 index de18d04d464..00000000000 --- a/integration/invalidation/invalidation-foreign/repo/foreignB/build.sc +++ /dev/null @@ -1,5 +0,0 @@ -import mill._ - -def taskB = T { - println("b") -} diff --git a/integration/invalidation/invalidation-foreign/test/src/ScriptsInvalidationForeignTests.scala b/integration/invalidation/invalidation-foreign/test/src/ScriptsInvalidationForeignTests.scala deleted file mode 100644 index a5d4c53734c..00000000000 --- a/integration/invalidation/invalidation-foreign/test/src/ScriptsInvalidationForeignTests.scala +++ /dev/null @@ -1,66 +0,0 @@ -package mill.integration - -import utest._ -import os.SubPath - -object ScriptsInvalidationForeignTests extends IntegrationTestSuite { - - override def buildPath: SubPath = os.sub / "foreignA" / "build.sc" - - def runTask(task: String): Vector[String] = { - val res = evalStdout(task) - assert(res.isSuccess) - res.out.linesIterator.map(_.trim).toVector - } - - val tests: Tests = Tests { - test("should handle foreign modules") { - test("first run") { - initWorkspace() - - val result = runTask("taskA") - - val expected = Seq("b", "a") - - assert(result == expected) - } - - test("second run modifying script") { - val oldContent = os.read(scriptSourcePath / buildPath) - val newContent = - oldContent.replace("""println("a")""", """println("a2")""") - os.write.over(workspacePath / buildPath, newContent) - - val result = runTask("taskA") - - val expected = Seq("a2") - - assert(result == expected) - } - } - test("should handle imports in higher level than top level") { - test("first run") { - initWorkspace() - - val result = runTask("taskD") - - val expected = Seq("c", "d") - - assert(result == expected) - } - - test("second run modifying script") { - val oldContent = os.read(scriptSourcePath / buildPath) - val newContent = - oldContent.replace("""println("d")""", """println("d2")""") - os.write.over(workspacePath / buildPath, newContent) - - val result = runTask("taskD") - - val expected = Seq("d2") - - assert(result == expected) - } - } - } -} diff --git "a/integration/invalidation/invalidation/repo/-#!+\342\206\222&%=~/inputSymbols.sc" "b/integration/invalidation/invalidation/repo/-#!+\342\206\222&%=~/module.sc" similarity index 100% rename from "integration/invalidation/invalidation/repo/-#!+\342\206\222&%=~/inputSymbols.sc" rename to "integration/invalidation/invalidation/repo/-#!+\342\206\222&%=~/module.sc" diff --git a/integration/invalidation/invalidation/repo/-#+&%.sc b/integration/invalidation/invalidation/repo/-#+&%/module.sc similarity index 100% rename from integration/invalidation/invalidation/repo/-#+&%.sc rename to integration/invalidation/invalidation/repo/-#+&%/module.sc diff --git a/integration/invalidation/invalidation/repo/a/inputA.sc b/integration/invalidation/invalidation/repo/a/module.sc similarity index 100% rename from integration/invalidation/invalidation/repo/a/inputA.sc rename to integration/invalidation/invalidation/repo/a/module.sc diff --git a/integration/invalidation/invalidation/repo/b/inputB.sc b/integration/invalidation/invalidation/repo/b/module.sc similarity index 100% rename from integration/invalidation/invalidation/repo/b/inputB.sc rename to integration/invalidation/invalidation/repo/b/module.sc diff --git a/integration/invalidation/invalidation/repo/build.sc b/integration/invalidation/invalidation/repo/build.sc index 38f6e6f131c..742a95e3d45 100644 --- a/integration/invalidation/invalidation/repo/build.sc +++ b/integration/invalidation/invalidation/repo/build.sc @@ -1,38 +1,33 @@ import mill._ -import $file.a.inputA -import $file.b.{inputB => inputBRenamed} -import $file.inputC + import $ivy.`org.scalaj::scalaj-http:2.4.2` -import $file.e.inputE -import $file.`-#!+→&%=~`.inputSymbols -import $file.`-#+&%` def task = T { - inputA.input() - inputBRenamed.input() - inputC.input() + build.a.input() + build.b.input() + build.c.input() } object module extends Module { def task = T { println("task") - inputA.input() - inputBRenamed.input() - inputC.input() + build.a.input() + build.b.input() + build.c.input() } } def taskE = T { println("taskE") - inputE.input() + build.e.input() } def taskSymbols = T { println("taskSymbols") - inputSymbols.input() + build.`-#!+→&%=~`.input() } def taskSymbolsInFile = T { println("taskSymbolsInFile") - `-#+&%`.module.input() + build.`-#+&%`.module.input() } diff --git a/integration/invalidation/invalidation/repo/inputC.sc b/integration/invalidation/invalidation/repo/c/module.sc similarity index 100% rename from integration/invalidation/invalidation/repo/inputC.sc rename to integration/invalidation/invalidation/repo/c/module.sc diff --git a/integration/invalidation/invalidation/repo/e/inputE.sc b/integration/invalidation/invalidation/repo/e/module.sc similarity index 53% rename from integration/invalidation/invalidation/repo/e/inputE.sc rename to integration/invalidation/invalidation/repo/e/module.sc index 9eb3d096b88..9cbd474eb57 100644 --- a/integration/invalidation/invalidation/repo/e/inputE.sc +++ b/integration/invalidation/invalidation/repo/e/module.sc @@ -1,7 +1,6 @@ import mill._ -import $file.^.a.inputA def input = T { println("e") - inputA.input() + build.a.input() } diff --git a/integration/src/mill/integration/IntegrationTestSuite.scala b/integration/src/mill/integration/IntegrationTestSuite.scala index b2224b610a5..67b08610ebc 100644 --- a/integration/src/mill/integration/IntegrationTestSuite.scala +++ b/integration/src/mill/integration/IntegrationTestSuite.scala @@ -128,7 +128,10 @@ abstract class IntegrationTestSuite extends TestSuite { // wrapper-folder inside the zip file, so copy the wrapper folder to the // destination instead of the folder containing the wrapper. - os.copy(scriptSourcePath, workspacePath) + // somehow os.copy does not properly preserve symlinks + // os.copy(scriptSourcePath, workspacePath) + os.proc("cp", "-R", scriptSourcePath, workspacePath).call() + os.remove.all(workspacePath / "out") workspacePath } diff --git a/main/define/src/mill/define/BaseModule.scala b/main/define/src/mill/define/BaseModule.scala index ea21cc8e966..36bcfb599df 100644 --- a/main/define/src/mill/define/BaseModule.scala +++ b/main/define/src/mill/define/BaseModule.scala @@ -1,5 +1,10 @@ package mill.define +import mill.api.PathRef +import mill.util.Watchable + +import scala.collection.mutable + object BaseModule { case class Implicit(value: BaseModule) } @@ -24,7 +29,8 @@ abstract class BaseModule( millFile0, caller ) - ) with Module { + ) with Module with BaseModule0 { + // A BaseModule should provide an empty Segments list to it's children, since // it is the root of the module tree, and thus must not include it's own // sourcecode.Name as part of the list, @@ -36,6 +42,40 @@ abstract class BaseModule( } +trait BaseModule0 extends Module { + implicit def millDiscover: Discover[_] + protected[mill] val watchedValues: mutable.Buffer[Watchable] = mutable.Buffer.empty[Watchable] + protected[mill] val evalWatchedValues: mutable.Buffer[Watchable] = mutable.Buffer.empty[Watchable] + + class Interp { + + def watchValue[T](v0: => T)(implicit fn: sourcecode.FileName, ln: sourcecode.Line): T = { + val v = v0 + val watchable = Watchable.Value( + () => v0.hashCode, + v.hashCode(), + fn.value + ":" + ln.value + ) + watchedValues.append(watchable) + v + } + + def watch(p: os.Path): os.Path = { + val watchable = Watchable.Path(PathRef(p)) + watchedValues.append(watchable) + p + } + + def watch0(w: Watchable): Unit = { + watchedValues.append(w) + } + + def evalWatch0(w: Watchable): Unit = { + evalWatchedValues.append(w) + } + } +} + abstract class ExternalModule(implicit millModuleEnclosing0: sourcecode.Enclosing, millModuleLine0: sourcecode.Line diff --git a/main/define/src/mill/define/BaseModuleTree.scala b/main/define/src/mill/define/BaseModuleTree.scala new file mode 100644 index 00000000000..802fd0575a4 --- /dev/null +++ b/main/define/src/mill/define/BaseModuleTree.scala @@ -0,0 +1,41 @@ +package mill.define + +/** + * Data structure containing the metadata of all [[BaseModule]]s defined in a Mill build, as well + * as pre-computed data structures for common lookup patterns during the target resolution process + */ +class BaseModuleTree(value: Seq[(Seq[String], BaseModule)]) { + private val prefixForClass0: Map[Class[_], Seq[String]] = value.map { case (p, m) => + (m.getClass, p) + }.toMap + def prefixForClass(cls: Class[_]): Option[Seq[String]] = prefixForClass0.get(cls) + + private val lookupByParent0: Map[Option[Seq[String]], Seq[(Seq[String], BaseModule)]] = + value.groupBy { + case (Nil, _) => None + case (xs :+ last, _) => Some(xs) + } + + def lookupByParent(parentPrefixOpt: Option[Seq[String]]): Seq[(Seq[String], BaseModule)] = + lookupByParent0.getOrElse(parentPrefixOpt, Nil) + + def rootModule: BaseModule = lookupByParent(None).head._2 + val allPossibleNames: Set[String] = + value.flatMap(_._2.millDiscover.value.values).flatMap(_._1).toSet +} +object BaseModuleTree { + def from(rootModules: Seq[BaseModule]): BaseModuleTree = { + new BaseModuleTree( + rootModules + .map { m => + val parts = m.getClass.getName match { + case s"build.$partString.package$$" => partString.split('.') + case s"build.${partString}_$$$last$$" => partString.split('.') + case _ => Array[String]() + } + + (parts, m) + } + ) + } +} diff --git a/main/eval/src/mill/eval/Evaluator.scala b/main/eval/src/mill/eval/Evaluator.scala index 8df7678817a..95cd15e1281 100644 --- a/main/eval/src/mill/eval/Evaluator.scala +++ b/main/eval/src/mill/eval/Evaluator.scala @@ -2,7 +2,7 @@ package mill.eval import mill.api.{CompileProblemReporter, DummyTestReporter, Result, TestReporter, Val} import mill.api.Strict.Agg -import mill.define.{BaseModule, Segments, Task} +import mill.define.{BaseModule, BaseModuleTree, Segments, Task} import mill.eval.Evaluator.{Results, formatFailing} import mill.util.{ColorLogger, MultiBiMap} @@ -15,7 +15,9 @@ import scala.util.DynamicVariable */ trait Evaluator { def baseLogger: ColorLogger - def rootModule: BaseModule + def rootModule: BaseModule = rootModules.head + def rootModules: Seq[BaseModule] = Seq(rootModule) + def baseModules: BaseModuleTree = BaseModuleTree.from(rootModules) def effectiveThreadCount: Int def outPath: os.Path def externalOutPath: os.Path diff --git a/main/eval/src/mill/eval/EvaluatorImpl.scala b/main/eval/src/mill/eval/EvaluatorImpl.scala index 4af6c8c30d9..56ca20c862a 100644 --- a/main/eval/src/mill/eval/EvaluatorImpl.scala +++ b/main/eval/src/mill/eval/EvaluatorImpl.scala @@ -17,7 +17,7 @@ private[mill] case class EvaluatorImpl( workspace: os.Path, outPath: os.Path, externalOutPath: os.Path, - rootModule: mill.define.BaseModule, + override val rootModules: Seq[mill.define.BaseModule], baseLogger: ColorLogger, classLoaderSigHash: Int, classLoaderIdentityHash: Int, diff --git a/main/eval/src/mill/eval/GroupEvaluator.scala b/main/eval/src/mill/eval/GroupEvaluator.scala index 61eba65d198..3ee01997a99 100644 --- a/main/eval/src/mill/eval/GroupEvaluator.scala +++ b/main/eval/src/mill/eval/GroupEvaluator.scala @@ -22,7 +22,7 @@ private[mill] trait GroupEvaluator { def workspace: os.Path def outPath: os.Path def externalOutPath: os.Path - def rootModule: mill.define.BaseModule + def rootModules: Seq[mill.define.BaseModule] def classLoaderSigHash: Int def classLoaderIdentityHash: Int def workerCache: mutable.Map[Segments, (Int, Val)] diff --git a/main/eval/src/mill/eval/Plan.scala b/main/eval/src/mill/eval/Plan.scala index ff50394b5c0..3a96a92d07e 100644 --- a/main/eval/src/mill/eval/Plan.scala +++ b/main/eval/src/mill/eval/Plan.scala @@ -32,7 +32,7 @@ private object Plan { Segments( segments.value.init ++ Seq(Segment.Label(tName + ".super")) ++ - t.ctx.enclosing.split("[.# ]").map(Segment.Label) + t.ctx.enclosing.split("[.# ]").filter(_ != "").map(Segment.Label) ) } Terminal.Labelled(t, augmentedSegments) diff --git a/main/resolve/src/mill/resolve/Resolve.scala b/main/resolve/src/mill/resolve/Resolve.scala index 1813a938807..adbbb0d85ab 100644 --- a/main/resolve/src/mill/resolve/Resolve.scala +++ b/main/resolve/src/mill/resolve/Resolve.scala @@ -3,9 +3,9 @@ package mill.resolve import mainargs.{MainData, TokenGrouping} import mill.define.{ BaseModule, + BaseModuleTree, Command, Discover, - ExternalModule, Module, NamedTask, Reflect, @@ -19,7 +19,7 @@ import mill.util.EitherOps object Resolve { object Segments extends Resolve[Segments] { private[mill] def handleResolved( - rootModule: BaseModule, + baseModules: BaseModuleTree, resolved: Seq[Resolved], args: Seq[String], selector: Segments, @@ -33,7 +33,7 @@ object Resolve { object Tasks extends Resolve[NamedTask[Any]] { private[mill] def handleResolved( - rootModule: BaseModule, + baseModules: BaseModuleTree, resolved: Seq[Resolved], args: Seq[String], selector: Segments, @@ -42,23 +42,25 @@ object Resolve { val taskList = resolved.map { case r: Resolved.Target => val instantiated = ResolveCore - .instantiateModule(rootModule, r.segments.init) + .instantiateModule(baseModules, r.segments.init) .flatMap(instantiateTarget(r, _)) instantiated.map(Some(_)) case r: Resolved.Command => val instantiated = ResolveCore - .instantiateModule(rootModule, r.segments.init) - .flatMap(instantiateCommand(rootModule, r, _, args, nullCommandDefaults)) + .instantiateModule0(baseModules, r.segments.init) + .flatMap { case (mod, rootMod) => + instantiateCommand(rootMod, r, mod, args, nullCommandDefaults) + } instantiated.map(Some(_)) case r: Resolved.Module => - ResolveCore.instantiateModule(rootModule, r.segments).flatMap { + ResolveCore.instantiateModule(baseModules, r.segments).flatMap { case value: TaskModule => val directChildrenOrErr = ResolveCore.resolveDirectChildren( - rootModule, + baseModules, value.getClass, Some(value.defaultCommandName()), value.millModuleSegments @@ -68,7 +70,13 @@ object Resolve { directChildren.head match { case r: Resolved.Target => instantiateTarget(r, value).map(Some(_)) case r: Resolved.Command => - instantiateCommand(rootModule, r, value, args, nullCommandDefaults).map(Some(_)) + instantiateCommand( + baseModules.rootModule, + r, + value, + args, + nullCommandDefaults + ).map(Some(_)) } ) case _ => Right(None) @@ -104,15 +112,17 @@ object Resolve { args: Seq[String], nullCommandDefaults: Boolean ) = { - ResolveCore.catchWrapException( - invokeCommand0( + ResolveCore.catchWrapException { + val invoked = invokeCommand0( p, r.segments.parts.last, rootModule.millDiscover.asInstanceOf[Discover[mill.define.Module]], args, nullCommandDefaults - ).head - ).flatten + ) + + invoked.head + }.flatten } private def invokeCommand0( @@ -181,7 +191,7 @@ object Resolve { trait Resolve[T] { private[mill] def handleResolved( - rootModule: BaseModule, + baseModules: BaseModuleTree, resolved: Seq[Resolved], args: Seq[String], segments: Segments, @@ -193,11 +203,18 @@ trait Resolve[T] { scriptArgs: Seq[String], selectMode: SelectMode ): Either[String, List[T]] = { - resolve0(rootModule, scriptArgs, selectMode) + resolve0(Seq(rootModule), scriptArgs, selectMode) + } + def resolve( + rootModules: Seq[BaseModule], + scriptArgs: Seq[String], + selectMode: SelectMode + ): Either[String, List[T]] = { + resolve0(rootModules, scriptArgs, selectMode) } private[mill] def resolve0( - baseModule: BaseModule, + baseModules: Seq[BaseModule], scriptArgs: Seq[String], selectMode: SelectMode ): Either[String, List[T]] = { @@ -205,8 +222,8 @@ trait Resolve[T] { val resolvedGroups = ParseArgs(scriptArgs, selectMode).flatMap { groups => val resolved = groups.map { case (selectors, args) => val selected = selectors.map { case (scopedSel, sel) => - resolveRootModule(baseModule, scopedSel).map { rootModule => - resolveNonEmptyAndHandle(args, sel, rootModule, nullCommandDefaults) + resolveRootModule(baseModules, scopedSel).map { rootModuleSels => + resolveNonEmptyAndHandle(args, rootModuleSels, sel, nullCommandDefaults) } } @@ -224,21 +241,21 @@ trait Resolve[T] { private[mill] def resolveNonEmptyAndHandle( args: Seq[String], + baseModules: BaseModuleTree, sel: Segments, - rootModule: BaseModule, nullCommandDefaults: Boolean ): Either[String, Seq[T]] = { - val rootResolved = ResolveCore.Resolved.Module(Segments(), rootModule.getClass) + val rootResolved = ResolveCore.Resolved.Module(Segments(), baseModules.rootModule.getClass) val resolved = ResolveCore.resolve( - rootModule = rootModule, + baseModules = baseModules, remainingQuery = sel.value.toList, current = rootResolved, querySoFar = Segments() ) match { case ResolveCore.Success(value) => Right(value) case ResolveCore.NotFound(segments, found, next, possibleNexts) => - val allPossibleNames = rootModule.millDiscover.value.values.flatMap(_._1).toSet + val allPossibleNames = baseModules.allPossibleNames Left(ResolveNotFoundHandler( selector = sel, segments = segments, @@ -252,30 +269,31 @@ trait Resolve[T] { resolved .map(_.toSeq.sortBy(_.segments.render)) - .flatMap(handleResolved(rootModule, _, args, sel, nullCommandDefaults)) + .flatMap(handleResolved(baseModules, _, args, sel, nullCommandDefaults)) } private[mill] def deduplicate(items: List[T]): List[T] = items private[mill] def resolveRootModule( - rootModule: BaseModule, + rootModules: Seq[BaseModule], scopedSel: Option[Segments] - ): Either[String, BaseModule] = { + ): Either[String, BaseModuleTree] = { scopedSel match { - case None => Right(rootModule) + case None => Right(BaseModuleTree.from(rootModules)) + case Some(scoping) => for { moduleCls <- - try Right(rootModule.getClass.getClassLoader.loadClass(scoping.render + "$")) + try Right(rootModules.head.getClass.getClassLoader.loadClass(scoping.render + "$")) catch { case e: ClassNotFoundException => Left("Cannot resolve external module " + scoping.render) } rootModule <- moduleCls.getField("MODULE$").get(moduleCls) match { - case rootModule: ExternalModule => Right(rootModule) - case _ => Left("Class " + scoping.render + " is not an external module") + case rootModule: BaseModule => Right(rootModule) + case _ => Left("Class " + scoping.render + " is not an BaseModule") } - } yield rootModule + } yield new BaseModuleTree(Seq((Nil, rootModule))) } } } diff --git a/main/resolve/src/mill/resolve/ResolveCore.scala b/main/resolve/src/mill/resolve/ResolveCore.scala index 823e9935acd..080bedbbdc3 100644 --- a/main/resolve/src/mill/resolve/ResolveCore.scala +++ b/main/resolve/src/mill/resolve/ResolveCore.scala @@ -64,7 +64,7 @@ private object ResolveCore { } def resolve( - rootModule: Module, + baseModules: BaseModuleTree, remainingQuery: List[Segment], current: Resolved, querySoFar: Segments @@ -74,7 +74,7 @@ private object ResolveCore { case head :: tail => def recurse(searchModules: Set[Resolved]): Result = { val (failures, successesLists) = searchModules - .map(r => resolve(rootModule, tail, r, querySoFar ++ Seq(head))) + .map(r => resolve(baseModules, tail, r, querySoFar ++ Seq(head))) .partitionMap { case s: Success => Right(s.value); case f: Failed => Left(f) } val (errors, notFounds) = failures.partitionMap { @@ -86,7 +86,7 @@ private object ResolveCore { else if (successesLists.flatten.nonEmpty) Success(successesLists.flatten) else notFounds.size match { case 1 => notFounds.head - case _ => notFoundResult(rootModule, querySoFar, current, head) + case _ => notFoundResult(baseModules, querySoFar, current, head) } } @@ -96,19 +96,30 @@ private object ResolveCore { case "__" => val self = Seq(Resolved.Module(m.segments, m.cls)) val transitiveOrErr = - resolveTransitiveChildren(rootModule, m.cls, None, current.segments, Nil) + resolveTransitiveChildren( + baseModules, + m.cls, + None, + current.segments, + Nil + ) transitiveOrErr.map(transitive => self ++ transitive) case "_" => - resolveDirectChildren(rootModule, m.cls, None, current.segments) + resolveDirectChildren( + baseModules, + m.cls, + None, + current.segments + ) case pattern if pattern.startsWith("__:") => val typePattern = pattern.split(":").drop(1) val self = Seq(Resolved.Module(m.segments, m.cls)) val transitiveOrErr = resolveTransitiveChildren( - rootModule, + baseModules, m.cls, None, current.segments, @@ -120,7 +131,7 @@ private object ResolveCore { case pattern if pattern.startsWith("_:") => val typePattern = pattern.split(":").drop(1) resolveDirectChildren( - rootModule, + baseModules, m.cls, None, current.segments, @@ -128,7 +139,12 @@ private object ResolveCore { ) case _ => - resolveDirectChildren(rootModule, m.cls, Some(singleLabel), current.segments) + resolveDirectChildren( + baseModules, + m.cls, + Some(singleLabel), + current.segments + ) } resOrErr match { @@ -138,23 +154,24 @@ private object ResolveCore { case (Segment.Cross(cross), m: Resolved.Module) => if (classOf[Cross[_]].isAssignableFrom(m.cls)) { - instantiateModule(rootModule, current.segments).flatMap { case c: Cross[_] => - catchWrapException( - if (cross == Seq("__")) for ((_, v) <- c.valuesToModules.toSeq) yield v - else if (cross.contains("_")) { - for { - (segments, v) <- c.segmentsToModules.toList - if segments.length == cross.length - if segments.zip(cross).forall { case (l, r) => l == r || r == "_" } - } yield v - } else { - val crossOrDefault = if (cross.isEmpty) { - // We want to select the default (first) crossValue - c.defaultCrossSegments - } else cross - c.segmentsToModules.get(crossOrDefault.toList).toSeq - } - ) + instantiateModule(baseModules, current.segments).flatMap { + case c: Cross[_] => + catchWrapException( + if (cross == Seq("__")) for ((_, v) <- c.valuesToModules.toSeq) yield v + else if (cross.contains("_")) { + for { + (segments, v) <- c.segmentsToModules.toList + if segments.length == cross.length + if segments.zip(cross).forall { case (l, r) => l == r || r == "_" } + } yield v + } else { + val crossOrDefault = if (cross.isEmpty) { + // We want to select the default (first) crossValue + c.defaultCrossSegments + } else cross + c.segmentsToModules.get(crossOrDefault.toList).toSeq + } + ) } match { case Left(err) => Error(err) case Right(searchModules) => @@ -165,68 +182,88 @@ private object ResolveCore { ) } - } else notFoundResult(rootModule, querySoFar, current, head) + } else notFoundResult(baseModules, querySoFar, current, head) - case _ => notFoundResult(rootModule, querySoFar, current, head) + case _ => notFoundResult(baseModules, querySoFar, current, head) } } } + def instantiateModule( + baseModules: BaseModuleTree, + segments: Segments + ): Either[String, Module] = + instantiateModule0(baseModules, segments).map(_._1) + + def instantiateModule0( + baseModules: BaseModuleTree, + segments: Segments + ): Either[String, (Module, BaseModule)] = { - def instantiateModule(rootModule: Module, segments: Segments): Either[String, Module] = { - segments.value.foldLeft[Either[String, Module]](Right(rootModule)) { - case (Right(current), Segment.Label(s)) => + segments.value.foldLeft[Either[String, (Module, BaseModule)]](Right(( + baseModules.rootModule, + baseModules.rootModule + ))) { + case (Right((current, currentRoot)), Segment.Label(s)) => assert(s != "_", s) resolveDirectChildren0( - rootModule, + baseModules, current.millModuleSegments, current.getClass, Some(s) ).flatMap { - case Seq((_, Some(f))) => f(current) + case Seq((_, Some(f))) => + val res = f(current) + res.map { + case b: BaseModule => + (b, b) + case b => + (b, currentRoot) + } case unknown => sys.error( s"Unable to resolve single child " + - s"rootModule: $rootModule, segments: ${segments.render}," + + s"rootModule: ${baseModules.rootModule}, segments: ${segments.render}," + s"current: $current, s: ${s}, unknown: $unknown" ) } - case (Right(current), Segment.Cross(vs)) => + case (Right((current, currentRoot)), Segment.Cross(vs)) => assert(!vs.contains("_"), vs) catchWrapException( current .asInstanceOf[Cross[_]] .segmentsToModules(vs.toList) - .asInstanceOf[Module] + .asInstanceOf[Module] -> currentRoot ) case (Left(err), _) => Left(err) } - } - def resolveTransitiveChildren( - rootModule: Module, - cls: Class[_], - nameOpt: Option[String], - segments: Segments - ): Either[String, Set[Resolved]] = - resolveTransitiveChildren(rootModule, cls, nameOpt, segments, Nil) + } def resolveTransitiveChildren( - rootModule: Module, + baseModules: BaseModuleTree, cls: Class[_], nameOpt: Option[String], segments: Segments, typePattern: Seq[String] ): Either[String, Set[Resolved]] = { - val direct = resolveDirectChildren(rootModule, cls, nameOpt, segments, typePattern) + val direct = + resolveDirectChildren(baseModules, cls, nameOpt, segments, typePattern) direct.flatMap { direct => for { - directTraverse <- resolveDirectChildren(rootModule, cls, nameOpt, segments, Nil) + directTraverse <- + resolveDirectChildren(baseModules, cls, nameOpt, segments, Nil) indirect0 = directTraverse .collect { case m: Resolved.Module => - resolveTransitiveChildren(rootModule, m.cls, nameOpt, m.segments, typePattern) + resolveTransitiveChildren( + baseModules, + m.cls, + nameOpt, + m.segments, + typePattern + ) } indirect <- EitherOps.sequence(indirect0).map(_.flatten) } yield direct ++ indirect @@ -266,15 +303,21 @@ private object ResolveCore { } def resolveDirectChildren( - rootModule: Module, + baseModules: BaseModuleTree, cls: Class[_], nameOpt: Option[String], segments: Segments ): Either[String, Set[Resolved]] = - resolveDirectChildren(rootModule, cls, nameOpt, segments, typePattern = Nil) + resolveDirectChildren( + baseModules, + cls, + nameOpt, + segments, + typePattern = Nil + ) def resolveDirectChildren( - rootModule: Module, + baseModules: BaseModuleTree, cls: Class[_], nameOpt: Option[String], segments: Segments, @@ -282,7 +325,7 @@ private object ResolveCore { ): Either[String, Set[Resolved]] = { val crossesOrErr = if (classOf[Cross[_]].isAssignableFrom(cls) && nameOpt.isEmpty) { - instantiateModule(rootModule: Module, segments).map { + instantiateModule(baseModules, segments).map { case cross: Cross[_] => for (item <- cross.items) yield { Resolved.Module(segments ++ Segment.Cross(item.crossSegments), item.cls) @@ -297,7 +340,7 @@ private object ResolveCore { classMatchesTypePred(typePattern)(c.cls) } - resolveDirectChildren0(rootModule, segments, cls, nameOpt, typePattern) + resolveDirectChildren0(baseModules, segments, cls, nameOpt, typePattern) .map( _.map { case (Resolved.Module(s, cls), _) => Resolved.Module(segments ++ s, cls) @@ -311,15 +354,15 @@ private object ResolveCore { } def resolveDirectChildren0( - rootModule: Module, + baseModules: BaseModuleTree, segments: Segments, cls: Class[_], nameOpt: Option[String] ): Either[String, Seq[(Resolved, Option[Module => Either[String, Module]])]] = - resolveDirectChildren0(rootModule, segments, cls, nameOpt, Nil) + resolveDirectChildren0(baseModules, segments, cls, nameOpt, Nil) def resolveDirectChildren0( - rootModule: Module, + baseModules: BaseModuleTree, segments: Segments, cls: Class[_], nameOpt: Option[String], @@ -329,7 +372,7 @@ private object ResolveCore { val modulesOrErr: Either[String, Seq[(Resolved, Option[Module => Either[String, Module]])]] = { if (classOf[DynamicModule].isAssignableFrom(cls)) { - instantiateModule(rootModule, segments).map { + instantiateModule(baseModules, segments).map { case m: DynamicModule => m.millModuleDirectChildren .filter(c => namePred(c.millModuleSegments.parts.last)) @@ -345,7 +388,20 @@ private object ResolveCore { ) } } else Right { - Reflect + val nestedModuleScObjects = + for { + prefix <- baseModules.prefixForClass(cls).toSeq + (prefix2, m2) <- baseModules.lookupByParent(Some(prefix)) + if nameOpt.isEmpty || nameOpt.contains(prefix2.last) + } yield ( + Resolved.Module( + Segments.labels(decode(prefix2.last)), + m2.getClass + ), + Some((_: Module) => Right(m2)) + ) + + val reflectMemberObjects = Reflect .reflectNestedObjects0[Module](cls, namePred) .filter { case (_, member) => @@ -372,6 +428,8 @@ private object ResolveCore { } ) } + + nestedModuleScObjects ++ reflectMemberObjects } } @@ -391,14 +449,19 @@ private object ResolveCore { } def notFoundResult( - rootModule: Module, + baseModules: BaseModuleTree, querySoFar: Segments, current: Resolved, next: Segment ): NotFound = { val possibleNexts = current match { case m: Resolved.Module => - resolveDirectChildren(rootModule, m.cls, None, current.segments).toOption.get.map( + resolveDirectChildren( + baseModules, + m.cls, + None, + current.segments + ).toOption.get.map( _.segments.value.last ) diff --git a/main/resolve/test/src/mill/main/ResolveTests.scala b/main/resolve/test/src/mill/main/ResolveTests.scala index 45d65bde8fd..309fecfa2cc 100644 --- a/main/resolve/test/src/mill/main/ResolveTests.scala +++ b/main/resolve/test/src/mill/main/ResolveTests.scala @@ -50,7 +50,7 @@ object ResolveTests extends TestSuite { def resolveTasks(selectorStrings: Seq[String]) = { Resolve.Tasks.resolve0( - module, + Seq(module), selectorStrings, SelectMode.Separated ) @@ -58,7 +58,7 @@ object ResolveTests extends TestSuite { def resolveMetadata(selectorStrings: Seq[String]) = { Resolve.Segments.resolve0( - module, + Seq(module), selectorStrings, SelectMode.Separated ).map(_.map(_.render)) diff --git a/main/src/mill/main/MainModule.scala b/main/src/mill/main/MainModule.scala index 308ef993713..e40a3e5df93 100644 --- a/main/src/mill/main/MainModule.scala +++ b/main/src/mill/main/MainModule.scala @@ -1,9 +1,8 @@ package mill.main import java.util.concurrent.LinkedBlockingQueue -import mill.define.Target +import mill.define.{BaseModule0, Command, NamedTask, Segments, Target, Task} import mill.api.{Ctx, Logger, PathRef, Result} -import mill.define.{Command, NamedTask, Segments, Task} import mill.eval.{Evaluator, EvaluatorPaths, Terminal} import mill.resolve.{Resolve, SelectMode} import mill.resolve.SelectMode.Separated @@ -19,7 +18,7 @@ object MainModule { targets: Seq[String], selectMode: SelectMode )(f: List[NamedTask[Any]] => T): Result[T] = { - Resolve.Tasks.resolve(evaluator.rootModule, targets, selectMode) match { + Resolve.Tasks.resolve(evaluator.rootModules, targets, selectMode) match { case Left(err) => Result.Failure(err) case Right(tasks) => Result.Success(f(tasks)) } @@ -60,39 +59,10 @@ object MainModule { * [[mill.define.Module]] containing all the default tasks that Mill provides: [[resolve]], * [[show]], [[inspect]], [[plan]], etc. */ -trait MainModule extends mill.define.Module { - protected[mill] val watchedValues: mutable.Buffer[Watchable] = mutable.Buffer.empty[Watchable] - protected[mill] val evalWatchedValues: mutable.Buffer[Watchable] = mutable.Buffer.empty[Watchable] - - object interp { - - def watchValue[T](v0: => T)(implicit fn: sourcecode.FileName, ln: sourcecode.Line): T = { - val v = v0 - val watchable = Watchable.Value( - () => v0.hashCode, - v.hashCode(), - fn.value + ":" + ln.value - ) - watchedValues.append(watchable) - v - } - - def watch(p: os.Path): os.Path = { - val watchable = Watchable.Path(PathRef(p)) - watchedValues.append(watchable) - p - } - - def watch0(w: Watchable): Unit = { - watchedValues.append(w) - } - - def evalWatch0(w: Watchable): Unit = { - evalWatchedValues.append(w) - } - } +trait MainModule extends BaseModule0 { - implicit def millDiscover: mill.define.Discover[_] + object interp extends Interp +// implicit def millDiscover: mill.define.Discover[_] /** * Show the mill version. @@ -108,7 +78,7 @@ trait MainModule extends mill.define.Module { */ def resolve(evaluator: Evaluator, targets: String*): Command[List[String]] = Target.command { val resolved = Resolve.Segments.resolve( - evaluator.rootModule, + evaluator.rootModules, targets, SelectMode.Multi ) @@ -138,7 +108,7 @@ trait MainModule extends mill.define.Module { private def plan0(evaluator: Evaluator, targets: Seq[String]) = { Resolve.Tasks.resolve( - evaluator.rootModule, + evaluator.rootModules, targets, SelectMode.Multi ) match { @@ -158,7 +128,7 @@ trait MainModule extends mill.define.Module { def path(evaluator: Evaluator, src: String, dest: String): Command[List[String]] = Target.command { val resolved = Resolve.Tasks.resolve( - evaluator.rootModule, + evaluator.rootModules, List(src, dest), SelectMode.Multi ) @@ -212,7 +182,8 @@ trait MainModule extends mill.define.Module { def rec(t: Task[_]): Seq[Segments] = { if (seen(t)) Nil // do nothing else t match { - case t: mill.define.Target[_] if evaluator.rootModule.millInternal.targets.contains(t) => + case t: mill.define.Target[_] + if evaluator.rootModules.exists(_.millInternal.targets.contains(t)) => Seq(t.ctx.segments) case _ => seen.add(t) @@ -237,11 +208,14 @@ trait MainModule extends mill.define.Module { if (t.asCommand.isEmpty) List() else { val mainDataOpt = evaluator - .rootModule - .millDiscover - .value - .get(t.ctx.enclosingCls) - .flatMap(_._2.find(_.name == t.ctx.segments.parts.last)) + .rootModules + .flatMap( + _.millDiscover + .value + .get(t.ctx.enclosingCls) + .flatMap(_._2.find(_.name == t.ctx.segments.parts.last)) + ) + .headOption mainDataOpt match { case Some(mainData) if mainData.renderedArgSigs.nonEmpty => @@ -360,7 +334,7 @@ trait MainModule extends mill.define.Module { Right(os.list(rootDir).filterNot(keepPath)) else mill.resolve.Resolve.Segments.resolve( - evaluator.rootModule, + evaluator.rootModules, targets, SelectMode.Multi ).map { ts => @@ -460,7 +434,7 @@ trait MainModule extends mill.define.Module { } Resolve.Tasks.resolve( - evaluator.rootModule, + evaluator.rootModules, targets, SelectMode.Multi ) match { diff --git a/main/src/mill/main/RootModule.scala b/main/src/mill/main/RootModule.scala index 25607d5e17f..d03d56dd012 100644 --- a/main/src/mill/main/RootModule.scala +++ b/main/src/mill/main/RootModule.scala @@ -12,30 +12,45 @@ import mill.define.{Caller, Discover, Segments} * defined at the top level of the `build.sc` and not nested in any other * modules. */ -abstract class RootModule()(implicit +abstract class RootModule(implicit baseModuleInfo: RootModule.Info, millModuleEnclosing0: sourcecode.Enclosing, millModuleLine0: sourcecode.Line, - millFile0: sourcecode.File -) extends mill.define.BaseModule(baseModuleInfo.millSourcePath0)( - millModuleEnclosing0, - millModuleLine0, - millFile0, - Caller(null) - ) with mill.main.MainModule { - - // Make BaseModule take the `millDiscover` as an implicit param, rather than - // defining it itself. That is so we can define it externally in the wrapper - // code and it have it automatically passed to both the wrapper BaseModule as - // well as any user-defined BaseModule that may be present, so the - // user-defined BaseModule can have a complete Discover[_] instance without - // needing to tediously call `override lazy val millDiscover = Discover[this.type]` - override lazy val millDiscover: Discover[this.type] = - baseModuleInfo.discover.asInstanceOf[Discover[this.type]] -} + millFile0: sourcecode.File, + ctx: mill.define.Ctx +) extends RootModule.Base(foreign0 = ctx.foreign) + +abstract class RootModuleForeign(implicit + baseModuleInfo: RootModule.Info, + millModuleEnclosing0: sourcecode.Enclosing, + millModuleLine0: sourcecode.Line, + millFile0: sourcecode.File, + ctx: mill.define.Ctx +) extends RootModule.Foreign(foreign0 = ctx.foreign) @internal object RootModule { + abstract class Base(foreign0: Option[Segments] = None)(implicit + baseModuleInfo: RootModule.Info, + millModuleEnclosing0: sourcecode.Enclosing, + millModuleLine0: sourcecode.Line, + millFile0: sourcecode.File + ) extends mill.define.BaseModule(baseModuleInfo.millSourcePath0, foreign0 = foreign0)( + millModuleEnclosing0, + millModuleLine0, + millFile0, + Caller(null) + ) with mill.main.MainModule { + + // Make BaseModule take the `millDiscover` as an implicit param, rather than + // defining it itself. That is so we can define it externally in the wrapper + // code and it have it automatically passed to both the wrapper BaseModule as + // well as any user-defined BaseModule that may be present, so the + // user-defined BaseModule can have a complete Discover[_] instance without + // needing to tediously call `override lazy val millDiscover = Discover[this.type]` + override lazy val millDiscover: Discover[this.type] = + baseModuleInfo.discover.asInstanceOf[Discover[this.type]] + } case class Info(millSourcePath0: os.Path, discover: Discover[_]) abstract class Foreign(foreign0: Option[Segments])(implicit @@ -48,8 +63,11 @@ object RootModule { millModuleLine0, millFile0, Caller(null) - ) with mill.main.MainModule { + ) { + + object interp extends Interp - override implicit lazy val millDiscover: Discover[this.type] = Discover[this.type] + override lazy val millDiscover: Discover[this.type] = + baseModuleInfo.discover.asInstanceOf[Discover[this.type]] } } diff --git a/main/src/mill/main/RunScript.scala b/main/src/mill/main/RunScript.scala index abefe803522..aa377e3386e 100644 --- a/main/src/mill/main/RunScript.scala +++ b/main/src/mill/main/RunScript.scala @@ -21,7 +21,7 @@ object RunScript { (Seq[Watchable], Either[String, Seq[(Any, Option[(TaskName, ujson.Value)])]]) ] = { val resolved = mill.eval.Evaluator.currentEvaluator.withValue(evaluator) { - Resolve.Tasks.resolve(evaluator.rootModule, scriptArgs, selectMode) + Resolve.Tasks.resolve(evaluator.rootModules, scriptArgs, selectMode) } for (targets <- resolved) yield evaluateNamed(evaluator, Agg.from(targets)) } diff --git a/main/src/mill/main/TokenReaders.scala b/main/src/mill/main/TokenReaders.scala index 50f7785313f..d3c37219fa1 100644 --- a/main/src/mill/main/TokenReaders.scala +++ b/main/src/mill/main/TokenReaders.scala @@ -13,7 +13,7 @@ object Tasks { def shortName = "" def read(s: Seq[String]): Either[String, Tasks[T]] = { Resolve.Tasks.resolve( - Evaluator.currentEvaluator.value.rootModule, + Evaluator.currentEvaluator.value.rootModules, s, SelectMode.Separated ).map(x => Tasks(x.asInstanceOf[Seq[mill.define.NamedTask[T]]])) diff --git a/main/testkit/src/mill/testkit/MillTestkit.scala b/main/testkit/src/mill/testkit/MillTestkit.scala index 45e691c5864..7dea683d357 100644 --- a/main/testkit/src/mill/testkit/MillTestkit.scala +++ b/main/testkit/src/mill/testkit/MillTestkit.scala @@ -100,7 +100,7 @@ trait MillTestKit { module.millSourcePath, outPath, outPath, - module, + Seq(module), logger, 0, 0, @@ -113,7 +113,7 @@ trait MillTestKit { def evalTokens(args: String*): Either[Result.Failing[_], (Seq[_], Int)] = { mill.eval.Evaluator.currentEvaluator.withValue(evaluator) { - Resolve.Tasks.resolve(evaluator.rootModule, args, SelectMode.Separated) + Resolve.Tasks.resolve(evaluator.rootModules, args, SelectMode.Separated) } match { case Left(err) => Left(Result.Failure(err)) case Right(resolved) => apply(resolved) diff --git a/runner/src/mill/runner/FileImportGraph.scala b/runner/src/mill/runner/FileImportGraph.scala index 17e517b7fa5..6660f10974f 100644 --- a/runner/src/mill/runner/FileImportGraph.scala +++ b/runner/src/mill/runner/FileImportGraph.scala @@ -96,31 +96,30 @@ object FileImportGraph { if seenRepo.find(_._1 == repo).isEmpty } seenRepo.addOne((repo, s)) (start, "_root_._", end) + case ImportTree(Seq(("$ivy", _), rest @ _*), mapping, start, end) => seenIvy.addAll(mapping.map(_._1)) (start, "_root_._", end) + case ImportTree(Seq(("$meta", _), rest @ _*), mapping, start, end) => millImport = true (start, "_root_._", end) - case ImportTree(Seq(("$file", _), rest @ _*), mapping, start, end) => + + case ImportTree(Seq(("$file", end0), rest @ _*), mapping, start, end) => val nextPaths = mapping.map { case (lhs, rhs) => nextPathFor(s, rest.map(_._1) :+ lhs) } fileImports.addAll(nextPaths) importGraphEdges(s) ++= nextPaths - if (rest.isEmpty) (start, "_root_._", end) - else { - val end = rest.last._2 - ( - start, - fileImportToSegments(projectRoot, nextPaths(0) / os.up, false) - .map(backtickWrap) - .mkString("."), - end - ) - } + val patchString = + (fileImportToSegments(projectRoot, nextPaths(0) / os.up, false) ++ Seq()) + .map(backtickWrap) + .mkString(".") + + (start, patchString, rest.lastOption.fold(end0)(_._2)) } val numNewLines = stmt.substring(start, end).count(_ == '\n') + stmt = stmt.patch(start, patchString + mill.util.Util.newLine * numNewLines, end - start) } @@ -129,6 +128,19 @@ object FileImportGraph { val useDummy = !os.exists(projectRoot / "build.sc") walkScripts(projectRoot / "build.sc", useDummy) + val buildFiles = os + .walk( + projectRoot, + followLinks = true, + skip = p => + p == projectRoot / "out" || + p == projectRoot / "mill-build" || + (os.isDir(p) && !os.exists(p / "module.sc")) + ) + .filter(_.last == "module.sc") + + buildFiles.foreach(walkScripts(_)) + new FileImportGraph( seenScripts.toMap, seenRepo.toSeq, @@ -154,6 +166,6 @@ object FileImportGraph { def fileImportToSegments(base: os.Path, s: os.Path, stripExt: Boolean): Seq[String] = { val rel = (s / os.up / (if (stripExt) s.baseName else s.last)).relativeTo(base) - Seq("millbuild") ++ Seq.fill(rel.ups)("^") ++ rel.segments + Seq("build") ++ Seq.fill(rel.ups)("^") ++ rel.segments } } diff --git a/runner/src/mill/runner/MillBuildBootstrap.scala b/runner/src/mill/runner/MillBuildBootstrap.scala index 7c92e1df259..bbf636daf30 100644 --- a/runner/src/mill/runner/MillBuildBootstrap.scala +++ b/runner/src/mill/runner/MillBuildBootstrap.scala @@ -4,9 +4,9 @@ import mill.util.{ColorLogger, PrefixLogger, Watchable} import mill.main.BuildInfo import mill.api.{PathRef, Val, internal} import mill.eval.Evaluator -import mill.main.{RootModule, RunScript} +import mill.main.RunScript import mill.resolve.SelectMode -import mill.define.{Discover, Segments} +import mill.define.{BaseModule, Discover, Segments} import java.net.URLClassLoader @@ -135,26 +135,32 @@ class MillBuildBootstrap( Nil, // We don't want to evaluate anything in this depth (and above), so we just skip creating an evaluator, // mainly because we didn't even constructed (compiled) it's classpath + None, None ) nestedState.add(frame = evalState, errorOpt = None) } else { - val validatedRootModuleOrErr = nestedState.frames.headOption match { + val validatedRootModulesOrErr = nestedState.frames.headOption match { case None => - getChildRootModule(nestedState.bootstrapModuleOpt.get, depth, projectRoot) + getChildRootModule(nestedState.bootstrapModuleOpt.get, depth, projectRoot).map(Seq(_)) case Some(nestedFrame) => - getRootModule(nestedFrame.classLoaderOpt.get, depth, projectRoot) + getRootModule( + nestedFrame.compileOutput.get, + nestedFrame.classLoaderOpt.get, + depth, + projectRoot + ) } - validatedRootModuleOrErr match { + validatedRootModulesOrErr match { case Left(err) => nestedState.add(errorOpt = Some(err)) - case Right(rootModule) => + case Right(rootModules) => val evaluator = makeEvaluator( prevFrameOpt.map(_.workerCache).getOrElse(Map.empty), nestedState.frames.headOption.map(_.scriptImportGraph).getOrElse(Map.empty), nestedState.frames.headOption.map(_.methodCodeHashSignatures).getOrElse(Map.empty), - rootModule, + rootModules, // We want to use the grandparent buildHash, rather than the parent // buildHash, because the parent build changes are instead detected // by analyzing the scriptImportGraph in a more fine-grained manner. @@ -178,7 +184,7 @@ class MillBuildBootstrap( if (depth != 0) { val retState = processRunClasspath( nestedState, - rootModule, + rootModules, evaluator, prevFrameOpt, prevOuterFrameOpt @@ -186,14 +192,14 @@ class MillBuildBootstrap( if (retState.errorOpt.isEmpty && depth == requestedDepth) { // TODO: print some message and indicate actual evaluated frame - val evalRet = processFinalTargets(nestedState, rootModule, evaluator) + val evalRet = processFinalTargets(nestedState, rootModules, evaluator) if (evalRet.errorOpt.isEmpty) retState else evalRet } else retState } else { - processFinalTargets(nestedState, rootModule, evaluator) + processFinalTargets(nestedState, rootModules, evaluator) } } } @@ -213,15 +219,15 @@ class MillBuildBootstrap( */ def processRunClasspath( nestedState: RunnerState, - rootModule: RootModule, + rootModules: Seq[BaseModule], evaluator: Evaluator, prevFrameOpt: Option[RunnerState.Frame], prevOuterFrameOpt: Option[RunnerState.Frame] ): RunnerState = { evaluateWithWatches( - rootModule, + rootModules, evaluator, - Seq("{runClasspath,scriptImportGraph,methodCodeHashSignatures}") + Seq("{runClasspath,compile,scriptImportGraph,methodCodeHashSignatures}") ) match { case (Left(error), evalWatches, moduleWatches) => val evalState = RunnerState.Frame( @@ -232,6 +238,7 @@ class MillBuildBootstrap( Map.empty, None, Nil, + None, Option(evaluator) ) @@ -240,6 +247,7 @@ class MillBuildBootstrap( case ( Right(Seq( Val(runClasspath: Seq[PathRef]), + Val(compile: mill.scalalib.api.CompilationResult), Val(scriptImportGraph: Map[os.Path, (Int, Seq[os.Path])]), Val(methodCodeHashSignatures: Map[String, Int]) )), @@ -282,6 +290,7 @@ class MillBuildBootstrap( methodCodeHashSignatures, Some(classLoader), runClasspath, + Some(compile.classes), Option(evaluator) ) @@ -296,7 +305,7 @@ class MillBuildBootstrap( */ def processFinalTargets( nestedState: RunnerState, - rootModule: RootModule, + rootModules: Seq[BaseModule], evaluator: Evaluator ): RunnerState = { @@ -305,7 +314,7 @@ class MillBuildBootstrap( val (evaled, evalWatched, moduleWatches) = Evaluator.allBootstrapEvaluators.withValue( Evaluator.AllBootstrapEvaluators(Seq(evaluator) ++ nestedState.frames.flatMap(_.evaluator)) ) { - evaluateWithWatches(rootModule, evaluator, targetsAndParams) + evaluateWithWatches(rootModules, evaluator, targetsAndParams) } val evalState = RunnerState.Frame( @@ -316,6 +325,7 @@ class MillBuildBootstrap( Map.empty, None, Nil, + None, Option(evaluator) ) @@ -326,7 +336,7 @@ class MillBuildBootstrap( workerCache: Map[Segments, (Int, Val)], scriptImportGraph: Map[os.Path, (Int, Seq[os.Path])], methodCodeHashSignatures: Map[String, Int], - rootModule: RootModule, + rootModules: Seq[BaseModule], millClassloaderSigHash: Int, millClassloaderIdentityHash: Int, depth: Int @@ -341,7 +351,7 @@ class MillBuildBootstrap( projectRoot, recOut(projectRoot, depth), recOut(projectRoot, depth), - rootModule, + rootModules, PrefixLogger(logger, "", tickerContext = bootLogPrefix), classLoaderSigHash = millClassloaderSigHash, classLoaderIdentityHash = millClassloaderIdentityHash, @@ -390,15 +400,15 @@ object MillBuildBootstrap { } def evaluateWithWatches( - rootModule: RootModule, + rootModules: Seq[BaseModule], evaluator: Evaluator, targetsAndParams: Seq[String] ): (Either[String, Seq[Any]], Seq[Watchable], Seq[Watchable]) = { - rootModule.evalWatchedValues.clear() + rootModules.foreach(_.evalWatchedValues.clear()) val evalTaskResult = RunScript.evaluateTasksNamed(evaluator, targetsAndParams, SelectMode.Separated) - val moduleWatched = rootModule.watchedValues.toVector - val addedEvalWatched = rootModule.evalWatchedValues.toVector + val moduleWatched = rootModules.flatMap(_.watchedValues).toVector + val addedEvalWatched = rootModules.flatMap(_.evalWatchedValues).toVector evalTaskResult match { case Left(msg) => (Left(msg), Nil, moduleWatched) @@ -412,24 +422,38 @@ object MillBuildBootstrap { } def getRootModule( + compileOut: PathRef, runClassLoader: URLClassLoader, depth: Int, projectRoot: os.Path - ): Either[String, RootModule] = { - val cls = runClassLoader.loadClass("millbuild.build$") - val rootModule0 = cls.getField("MODULE$").get(cls).asInstanceOf[RootModule] - getChildRootModule(rootModule0, depth, projectRoot) + ): Either[String, Seq[BaseModule]] = { + + val packageClassNames = os + .walk(compileOut.path) + .map(_.relativeTo(compileOut.path).segments) + .collect { case "build" +: rest :+ "package$.class" => + ("build" +: rest :+ "package$").mkString(".") + } + + val packageClasses = packageClassNames.map(runClassLoader.loadClass(_)) + val rootModule0s = + packageClasses.map(cls => cls.getField("MODULE$").get(cls).asInstanceOf[BaseModule]) + val children = rootModule0s.map(getChildRootModule(_, depth, projectRoot)) + children.flatMap(_.left.toOption) match { + case Nil => Right[String, Seq[BaseModule]](children.map(_.toOption.get)) + case errors => Left(errors.mkString("\n")) + } } def getChildRootModule( - rootModule0: RootModule, + rootModule0: BaseModule, depth: Int, projectRoot: os.Path - ): Either[String, RootModule] = { + ): Either[String, BaseModule] = { - val childRootModules: Seq[RootModule] = rootModule0 + val childRootModules: Seq[BaseModule] = rootModule0 .millInternal - .reflectNestedObjects[RootModule]() + .reflectNestedObjects[BaseModule]() val rootModuleOrErr = childRootModules match { case Seq() => Right(rootModule0) diff --git a/runner/src/mill/runner/MillBuildRootModule.scala b/runner/src/mill/runner/MillBuildRootModule.scala index d26e36a8865..dc8e8030476 100644 --- a/runner/src/mill/runner/MillBuildRootModule.scala +++ b/runner/src/mill/runner/MillBuildRootModule.scala @@ -10,7 +10,7 @@ import mill.util.Util.millProjectModule import mill.scalalib.api.Versions import pprint.Util.literalize import FileImportGraph.backtickWrap -import mill.main.BuildInfo +import mill.main.{BuildInfo, RootModule} import scala.collection.immutable.SortedMap import scala.util.Try @@ -30,7 +30,7 @@ import scala.util.Try class MillBuildRootModule()(implicit baseModuleInfo: RootModule.Info, millBuildRootModuleInfo: MillBuildRootModule.Info -) extends RootModule() with ScalaModule { +) extends RootModule.Base() with ScalaModule { override def bspDisplayName0: String = millBuildRootModuleInfo .projectRoot .relativeTo(millBuildRootModuleInfo.topLevelProjectRoot) @@ -277,7 +277,7 @@ object MillBuildRootModule { topLevelProjectRoot0: os.Path, projectRoot: os.Path, enclosingClasspath: Seq[os.Path] - )(implicit baseModuleInfo: RootModule.Info) extends RootModule { + )(implicit baseModuleInfo: RootModule.Info) extends RootModule.Base { implicit private def millBuildRootModuleInfo: Info = MillBuildRootModule.Info( enclosingClasspath, @@ -315,11 +315,15 @@ object MillBuildRootModule { val relative = scriptSource.path.relativeTo(base) val dest = targetDest / FileImportGraph.fileImportToSegments(base, scriptSource.path, false) + val pkg = FileImportGraph.fileImportToSegments(base, scriptSource.path, true).dropRight(1) + val specialNames = Set("build", "module") val newSource = MillBuildRootModule.top( - relative, + specialNames(scriptSource.path.baseName), + if (specialNames(scriptSource.path.baseName)) relative.segments.init + else relative.segments.init :+ relative.baseName, scriptSource.path / os.up, - FileImportGraph.fileImportToSegments(base, scriptSource.path, true).dropRight(1), - scriptSource.path.baseName, + if (specialNames(scriptSource.path.baseName)) pkg.dropRight(1) else pkg, + if (specialNames(scriptSource.path.baseName)) pkg.last else scriptSource.path.baseName, enclosingClasspath, millTopLevelProjectRoot, scriptSource.path @@ -332,7 +336,8 @@ object MillBuildRootModule { } def top( - relative: os.RelPath, + isBuildOrModuleSc: Boolean, + segs: Seq[String], base: os.Path, pkg: Seq[String], name: String, @@ -340,48 +345,55 @@ object MillBuildRootModule { millTopLevelProjectRoot: os.Path, originalFilePath: os.Path ): String = { + val segsList = segs.map(pprint.Util.literalize(_)).mkString(", ") val superClass = - if (pkg.size <= 1 && name == "build") "_root_.mill.main.RootModule" - else { - // Computing a path in "out" that uniquely reflects the location - // of the foreign module relatively to the current build. - - // Encoding the number of `/..` - val ups = if (relative.ups > 0) Seq(s"up-${relative.ups}") else Seq() - val segs = - Seq("foreign-modules") ++ - ups ++ - relative.segments.init ++ - Seq(relative.segments.last.stripSuffix(".sc")) - - val segsList = segs.map(pprint.Util.literalize(_)).mkString(", ") + if (name == "build") { + s"_root_.mill.main.RootModule.Base(Some(_root_.mill.define.Segments.labels($segsList)))" + } else { s"_root_.mill.main.RootModule.Foreign(Some(_root_.mill.define.Segments.labels($segsList)))" } + val imports = "" +// if (name == "build") "import mill.main.RootModule" +// else "import mill.main.{RootModuleForeign => RootModule}" + val miscInfoName = s"MiscInfo_$name" - s"""package ${pkg.map(backtickWrap).mkString(".")} - | - |import _root_.mill.runner.MillBuildRootModule - | - |object ${backtickWrap(miscInfoName)} { - | implicit lazy val millBuildRootModuleInfo: _root_.mill.runner.MillBuildRootModule.Info = _root_.mill.runner.MillBuildRootModule.Info( - | ${enclosingClasspath.map(p => literalize(p.toString))}.map(_root_.os.Path(_)), - | _root_.os.Path(${literalize(base.toString)}), - | _root_.os.Path(${literalize(millTopLevelProjectRoot.toString)}), - | ) - | implicit lazy val millBaseModuleInfo: _root_.mill.main.RootModule.Info = _root_.mill.main.RootModule.Info( - | millBuildRootModuleInfo.projectRoot, - | _root_.mill.define.Discover[${backtickWrap(name)}] - | ) - |} - |import ${backtickWrap(miscInfoName)}.{millBuildRootModuleInfo, millBaseModuleInfo} - |object ${backtickWrap(name)} extends ${backtickWrap(name)} - |class ${backtickWrap(name)} extends $superClass { - | - |//MILL_ORIGINAL_FILE_PATH=${originalFilePath} - |//MILL_USER_CODE_START_MARKER - |""".stripMargin + val pkgLine = pkg.map(p => "package " + backtickWrap(p)).mkString("\n") + + if (isBuildOrModuleSc) { + s"""$pkgLine + | + |import _root_.mill.runner.MillBuildRootModule + |$imports + | + |object ${backtickWrap(miscInfoName)} { + | implicit lazy val millBuildRootModuleInfo: _root_.mill.runner.MillBuildRootModule.Info = _root_.mill.runner.MillBuildRootModule.Info( + | ${enclosingClasspath.map(p => literalize(p.toString))}.map(_root_.os.Path(_)), + | _root_.os.Path(${literalize(base.toString)}), + | _root_.os.Path(${literalize(millTopLevelProjectRoot.toString)}), + | ) + | implicit lazy val millBaseModuleInfo: _root_.mill.main.RootModule.Info = _root_.mill.main.RootModule.Info( + | millBuildRootModuleInfo.projectRoot, + | _root_.mill.define.Discover[${backtickWrap(name + "_")}] + | ) + |} + |import ${backtickWrap(miscInfoName)}.{millBuildRootModuleInfo, millBaseModuleInfo} + |package object ${backtickWrap(name)} extends ${backtickWrap(name + "_")} + |import ${backtickWrap(name)}._ + |class ${backtickWrap(name + "_")} extends $superClass { + | + |//MILL_ORIGINAL_FILE_PATH=${originalFilePath} + |//MILL_USER_CODE_START_MARKER + |""".stripMargin + } else { + s"""$pkgLine + |object ${backtickWrap(name)} { + | + |//MILL_ORIGINAL_FILE_PATH=${originalFilePath} + |//MILL_USER_CODE_START_MARKER + |""".stripMargin + } } val bottom = "\n}" diff --git a/runner/src/mill/runner/RunnerState.scala b/runner/src/mill/runner/RunnerState.scala index 4d1d30a2823..cb120197246 100644 --- a/runner/src/mill/runner/RunnerState.scala +++ b/runner/src/mill/runner/RunnerState.scala @@ -29,7 +29,7 @@ import mill.main.RootModule */ @internal case class RunnerState( - bootstrapModuleOpt: Option[RootModule], + bootstrapModuleOpt: Option[RootModule.Base], frames: Seq[RunnerState.Frame], errorOpt: Option[String] ) { @@ -62,6 +62,7 @@ object RunnerState { methodCodeHashSignatures: Map[String, Int], classLoaderOpt: Option[RunnerState.URLClassLoader], runClasspath: Seq[PathRef], + compileOutput: Option[PathRef], evaluator: Option[Evaluator] ) { @@ -102,7 +103,7 @@ object RunnerState { ) implicit val loggedRw: ReadWriter[Logged] = macroRW - def empty: Frame = Frame(Map.empty, Nil, Nil, Map.empty, Map.empty, None, Nil, null) + def empty: Frame = Frame(Map.empty, Nil, Nil, Map.empty, Map.empty, None, Nil, None, null) } } diff --git a/scalalib/src/mill/scalalib/Dependency.scala b/scalalib/src/mill/scalalib/Dependency.scala index 097c2a1c221..f2b6eca79c4 100644 --- a/scalalib/src/mill/scalalib/Dependency.scala +++ b/scalalib/src/mill/scalalib/Dependency.scala @@ -17,8 +17,8 @@ object Dependency extends ExternalModule { DependencyUpdatesImpl( ev, implicitly, - ev.rootModule, - ev.rootModule.millDiscover, + ev.rootModules.head, + ev.rootModules.head.millDiscover, allowPreRelease ) } diff --git a/scalalib/src/mill/scalalib/GenIdeaImpl.scala b/scalalib/src/mill/scalalib/GenIdeaImpl.scala index e9b49bd8e11..b4e8ee0fcc1 100755 --- a/scalalib/src/mill/scalalib/GenIdeaImpl.scala +++ b/scalalib/src/mill/scalalib/GenIdeaImpl.scala @@ -101,7 +101,7 @@ case class GenIdeaImpl( } val buildDepsPaths = Classpath - .allJars(evaluator.rootModule.getClass.getClassLoader) + .allJars(evaluator.rootModules.getClass.getClassLoader) .map(url => os.Path(java.nio.file.Paths.get(url.toURI))) def resolveTasks: Seq[Task[ResolvedModule]] = modules.map {