Swift MCP Gui

Swift MCP GUI Server

A Model Context Protocol (MCP) server that allows controlling macOS through SwiftAutoGUI. This server provides tools for programmatically controlling the mouse and keyboard through MCP clients.

Requirements

• macOS 27.0 or later (required by SwiftAutoGUI 1.0.0)
• Swift 6.2 or later
• Xcode 27.0 or later

Installation

Homebrew (recommended)

Install via the NakaokaRei/tap:

brew install NakaokaRei/tap/swift-mcp-gui

Then point your MCP client at the installed binary:

{
  "mcpServers": {
    "swift-mcp-gui": {
      "command": "/opt/homebrew/bin/swift-mcp-gui"
    }
  }
}

On Intel Macs the path is /usr/local/bin/swift-mcp-gui. To upgrade later, run brew upgrade swift-mcp-gui.

From source

1. Clone this repository: bash git clone https://github.com/NakaokaRei/swift-mcp-gui.git cd swift-mcp-gui
2. Install bash swift package experimental-install
3. Add command to your MCP client. `json { "mcpServers" : { "swift-mcp-gui" : { "command" : "/Users/USERNAME/.swiftpm/bin/swift-mcp-gui" } } }

## Available Tools

The server provides the following tools for controlling macOS:

### 1. Mouse Movement
- Tool name: `moveMouse`
- Input:
  - `x`: number (x-coordinate) - accepts integers, doubles, or string representations
  - `y`: number (y-coordinate) - accepts integers, doubles, or string representations
- Moves the mouse cursor to the specified coordinates

### 2. Mouse Clicks
- Tool name: `mouseClick`
- Input:
  - `button`: string ("left" or "right")
- Performs a mouse click at the current cursor position

### 3. Keyboard Input
- Tool name: `sendKeys`
- Input:
  - `keys`: array of strings (key names)
- Sends keyboard shortcuts or key combinations
- Example keys: "command", "control", "option", "shift", "return", "space", "a", "1", etc.

### 4. Scrolling
- Tool name: `scroll`
- Input:
  - `direction`: string ("up", "down", "left", "right")
  - `clicks`: number (number of scroll clicks)
- Performs scrolling in the specified direction

### 5. Screen Size
- Tool name: `getScreenSize`
- Returns the main screen dimensions (width and height)

### 6. Pixel Color
- Tool name: `getPixelColor`
- Input:
  - `x`: number (x-coordinate) - accepts integers, doubles, or string representations
  - `y`: number (y-coordinate) - accepts integers, doubles, or string representations
- Returns the RGBA color values (0-255) of the pixel at the specified coordinates

### 7. Capture Screen
- Tool name: `captureScreen`
- Input:
  - `quality`: number (optional, 0.0-1.0, default: 0.5) - JPEG compression quality
  - `scale`: number (optional, 0.1-1.0, default: 0.25) - Scale factor for image size
  - `output`: string (optional, "path" or "image", default: "path") - Output format
- `output: "path"` (default): Saves to a temporary file and returns the file path with dimensions. Reduces token consumption.
- `output: "image"`: Returns inline image content for AI vision (e.g. Claude)

### 8. Capture Region
- Tool name: `captureRegion`
- Input:
  - `x`: number (x-coordinate of the region)
  - `y`: number (y-coordinate of the region)
  - `width`: number (width of the region)
  - `height`: number (height of the region)
  - `quality`: number (optional, 0.0-1.0, default: 0.5) - JPEG compression quality
  - `scale`: number (optional, 0.1-1.0, default: 0.25) - Scale factor for image size
  - `output`: string (optional, "path" or "image", default: "path") - Output format
- `output: "path"` (default): Saves to a temporary file and returns the file path with dimensions. Reduces token consumption.
- `output: "image"`: Returns inline image content for AI vision (e.g. Claude)

### 9. Save Screenshot
- Tool name: `saveScreenshot`
- Input:
  - `filename`: string (path to save the screenshot)
  - `x`: number (optional, x-coordinate of the region)
  - `y`: number (optional, y-coordinate of the region)
  - `width`: number (optional, width of the region)
  - `height`: number (optional, height of the region)
  - `quality`: number (optional, 0.0-1.0, default: 0.1) - JPEG compression quality
  - `scale`: number (optional, 0.1-1.0, default: 0.25) - Scale factor for image size
- Captures the screen or a region and saves it to a file
- File format is determined by the filename extension (.jpg, .jpeg, .png)
- Quality parameter only affects JPEG files

### 10. Execute AppleScript
- Tool name: `executeAppleScript`
- Input:
  - `script`: string (AppleScript code to execute)
- Executes AppleScript code directly and returns the result
- Returns "AppleScript Result: <result>" if the script returns a value
- Returns "AppleScript executed successfully (no result returned)" if the script completes without returning a value

### 11. Execute AppleScript File
- Tool name: `executeAppleScriptFile`
- Input:
  - `path`: string (path to the AppleScript file)
- Executes an AppleScript from a file and returns the result
- Returns "AppleScript Result: <result>" if the script returns a value
- Returns "AppleScript file executed successfully (no result returned): " if the script completes without returning a value

## Security Considerations

This server requires full accessibility permissions in System Preferences to control your mouse and keyboard. Be careful when running it and only connect trusted MCP clients.

## License

MIT License