Hello @Jude Park - and thanks for providing such a detailed description.
When you package your desktop Java (Kotlin Compose) application as an MSIX and run it, the JVM (and libraries) attempt to use Java NIO’s selector mechanism.
Internally, on Windows, the default selector provider may use internal “pipe” or “socketpair” type constructs (in your case sun.nio.ch.UnixDomainSockets.connect0) for inter-thread/async operations.
Under a normal Win32 environment (MSI) that works fine. However, when installed as an MSIX and executed inside an AppContainer sandbox, certain operations may be restricted or behave differently — in particular loopback, local socket creation, or Unix domain socket semantics.
Because the stack fails at the native method connect0 with “Invalid argument: connect”, it strongly suggests the AppContainer environment is disallowing or altering behavior of internal Unix domain socket creation.
According to Microsoft’s documentation:
“An AppContainer app’s process and its child processes run inside a lightweight app container where they can access only the resources that are specifically granted to them.”
and
“Network isolation: Isolating the application from network resources beyond those specifically allocated, AppContainer prevents the application from escaping its environment.”
Unfortunately, Microsoft does not publish any explicit guarantee that Java NIO selectors or Unix-domain sockets are fully supported under MSIX AppContainer contexts. There are known restrictions on loopback, named pipes, and UWP-style network access.
“Network drop events from the AppContainer loopback block filter origin occur when localhost loopback isn’t enabled properly for the Universal Windows Platform (UWP) app.”
So it appears your JVM’s default selector provider is using a native path that the sandbox blocks.
Given your scenario (desktop app needing local WebSocket, HTTP, or OAuth callback servers on localhost), here are some approaches:
1. Switch selector provider or forced fallback
Provide the JVM argument:
-Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.WindowsSelectorProvider
You mentioned trying this — make sure it’s actually taking effect (log SelectorProvider.provider() at runtime).
Alternatively, consider forcing a simpler selector implementation (e.g. NIO select/poll) instead of one relying on Unix domain sockets.
You may also patch or override libraries like Ktor CIO or Java-WebSocket to use plain TCP/IP (127.0.0.1) rather than Unix domain sockets.
2. Use blocking I/O instead of NIO
If your local server doesn’t need high-throughput async, consider refactoring to java.net.ServerSocket / Socket or a blocking I/O library.
That can bypass the NIO selector path that triggers the sandbox restriction.
3. Use a networking stack known to be MSIX-compatible
Pick a library that explicitly works under AppContainer conditions and uses normal loopback TCP/IP. Confirm in documentation that it doesn’t rely on native Unix domain sockets or shared-memory pipes.
Also ensure you declare the proper network capabilities in your manifest:
“The three networking capabilities are
internetClient,internetClientServer, andprivateNetworkClientServer.”
4. Consider distributing via MSI (non-AppContainer)
If you determine that MSIX AppContainer imposes hard limitations on your networking model and refactoring is too costly, you can deliver the app via MSI with full trust (no sandbox).
Microsoft Store primarily supports Win32 desktop apps via MSIX (“Desktop Bridge”), but MSI distribution is still possible outside the Store. Consult official Store guidance or Partner Center support for confirmation.
“You can declare your packaged desktop app’s
uap10:TrustLevelasappContainer(sandboxed) ormediumIL(full-trust).”
What I’d recommend next
- Create a minimal Java test (e.g.
Selector.open(),HttpClient.newHttpClient(), and a loopback socket) inside your MSIX-packaged app to confirm whichSelectorProvideris active and capture any stack traces. - If the provider defaults to Unix domain sockets, retry with
WindowsSelectorProvider. - Evaluate how much effort it would take to switch your local servers to use TCP loopback instead of NIO/Unix domain sockets.
- If that’s not practical, explore packaging as full-trust (
runFullTrust) or fallback to MSI distribution. - When following up with Microsoft Support, include:
- SelectorProvider info and stack trace
- AppContainer trust level
- Network capabilities declared in manifest
This will help confirm whether AppContainer restrictions on Unix domain sockets apply broadly to Java apps or only to your setup.
Answers to your three questions
- Is Java NIO Selector blocked in MSIX AppContainer? Some internal NIO operations (especially Unix domain socket usage) appear to fail inside the AppContainer. Microsoft documentation confirms that AppContainer isolates both filesystem and network access, and loopback requires explicit enablement. (MSIX AppContainer apps) (Filter origin documentation
- What’s the recommended approach for Java desktop apps needing local servers?
Use plain TCP/IP loopback (
127.0.0.1) with blocking I/O or a compatible async library. Package withrunFullTrustif the app requires local network access, and avoid Unix domain sockets or selector providers restricted by AppContainer isolation. (AppContainer isolation) - Can you distribute MSI through Microsoft Store? Win32 desktop apps are primarily accepted through MSIX packaging. Pure MSI submissions are not directly supported for Store listing, but you can distribute MSI installers outside the Store. (How packaged desktop apps run on Windows)
Hope this helps! If my answer was useful, please follow the instructions here so others with the same problem can benefit as well.