Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

externalPackages doesn't work in monorepos #20

Open
IAlphaOmegaI opened this issue Nov 21, 2024 · 2 comments
Open

externalPackages doesn't work in monorepos #20

IAlphaOmegaI opened this issue Nov 21, 2024 · 2 comments

Comments

@IAlphaOmegaI
Copy link

External Packages Fail in Monorepos Due to Dependency Hoisting - Major DX Issue

Description
There's a significant issue when using Convex's externalPackages for node envs feature within monorepos. The current implementation doesn't account for monorepo package management, where dependencies are typically hoisted to the root. This affects ALL external packages, with Sharp being a prime example in my case.

Problem Scope
This is NOT just a Sharp-specific issue - it affects any external package used with Convex in a monorepo setup.
The externalPackages feature expects packages in the local node_modules, and since monorepos hoist the dependecies to the root, convex is unable to find them.

Current Behavior
When using ANY external package in a monorepo:

  1. Dependencies are hoisted to root node_modules (standard monorepo behavior)
  2. Convex can't locate the package even when it's in root package.json
  3. Deployment fails due to missing dependencies
  4. Adding to root package.json is insufficient - must also add to the Convex app's package.json

Example with Sharp

monorepo/
  ├─ node_modules/ (where packages actually live)
  ├─ apps/
  │  ├─ convex-app/
  │  │  ├─ node_modules/ (where Convex expects packages)
  │  │  ├─ package.json (needs duplicate declaration)

Current Workaround (Hacky)

  1. Create manual node_modules inside the Convex app directory
  2. Physically copy package files from root node_modules
  3. Add duplicate package entries in both root and app package.json
  4. Configure as externalPackage

Suggested Solutions

  1. Short-term: Document this limitation clearly in the externalPackages docs
  2. Medium-term: Add monorepo detection and automatic package location resolution
  3. Long-term: Redesign the externalPackages feature to work with hoisted dependencies

Environment

  • Package Managers Affected: pnpm, yarn, npm, bun (all with workspaces)
  • Monorepo Tools: Turborepo, Nx, others
  • Convex Version: ^1.14.4
@thomasballinger
Copy link
Collaborator

@IAlphaOmegaI thanks for submitting, sorry this slipped through the cracks for a month. If you have a paid Convex plan feel free to email support@convex.dev if you're blocked on something like this in the future.

What's your status with this currently, how's the workaround going?

There are several different ways monorepos can be set up, we use Rush which doesn't use a shared package.json at the root, but yeah the root style is more common.

Sounds like the convex CLI needs to implement the same package resolution behavior as Node.js, walking up directories to the first ancestor with a node_modules directory with this directory.

@IAlphaOmegaI
Copy link
Author

@IAlphaOmegaI thanks for submitting, sorry this slipped through the cracks for a month. If you have a paid Convex plan feel free to email support@convex.dev if you're blocked on something like this in the future.

What's your status with this currently, how's the workaround going?

There are several different ways monorepos can be set up, we use Rush which doesn't use a shared package.json at the root, but yeah the root style is more common.

Sounds like the convex CLI needs to implement the same package resolution behavior as Node.js, walking up directories to the first ancestor with a node_modules directory with this directory.

The workaround works great honestly, it's not that hard to maintain and it does the job well. Since I don't use a big number of external dependencies, it is not hard for me to cut and paste from monorepo node_modules to apps/provider (where my convex app is) node_modules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants