AnyCable-Go configuration

You can configure AnyCable-Go via CLI options, e.g.:

$ anycable-go --rpc_host=localhost:50051 --headers=cookie \
              --redis_url=redis://localhost:6379/5 --redis_channel=__anycable__ \
              --host=localhost --port=8080Copy to clipboardErrorCopied

Or via the corresponding environment variables (i.e. ANYCABLE_RPC_HOST, ANYCABLE_REDIS_URL, etc.).

Primary settings

Here is the list of the most commonly used configuration parameters.

NOTE: To see all available options run anycable-go -h.

--host, --port (ANYCABLE_HOST, ANYCABLE_PORT or PORT)

Server host and port (default: "localhost:8080").

--rpc_host (ANYCABLE_RPC_HOST)

RPC service address (default: "localhost:50051").

--rpc_impl (ANYCABLE_RPC_IMPL)

RPC implementation to use, HTTP or gRPC (default: inferred from the rpc_host value, so no need to specify manually in most cases). See below for more details on HTTP RPC.

--path (ANYCABLE_PATH)

WebSocket endpoint path (default: "/cable").

You can specify multiple paths separated by commas.

You can also use wildcards (at the end of the paths) or path placeholders:

anycable-go --path="/cable,/admin/cable/*,/accounts/{tenant}/cable"Copy to clipboardErrorCopied

--headers (ANYCABLE_HEADERS)

Comma-separated list of headers to proxy to RPC (default: "cookie").

--proxy-cookies (ANYCABLE_PROXY_COOKIES)

Comma-separated list of cookies to proxy to RPC (default: all cookies).

--allowed_origins (ANYCABLE_ALLOWED_ORIGINS)

Comma-separated list of hostnames to check the Origin header against during the WebSocket Upgrade. Supports wildcards, e.g., --allowed_origins=*.evilmartians.io,www.evilmartians.com.

--broadcast_adapter (ANYCABLE_BROADCAST_ADAPTER, default: redis)

Broadcasting adapter to use. Available options: redis (default), redisx, nats, and http.

When HTTP adapter is used, AnyCable-Go accepts broadcasting requests on :8090/_broadcast.

You can also enable multiple adapters at once by specifying them separated by commas.

--broker (ANYCABLE_BROKER, default: none)

Broker adapter to use.

--pubsub (ANYCABLE_PUBSUB, default: none)

Pub/Sub adapter to use to distribute broadcasted messages within the cluster (when non-distributed broadcasting adapter is used). Required for broker.

--http_broadcast_port (ANYCABLE_HTTP_BROADCAST_PORT, default: 8090)

You can specify on which port to receive broadcasting requests (NOTE: it could be the same port as the main HTTP server listens to).

--http_broadcast_secret (ANYCABLE_HTTP_BROADCAST_SECRET)

Authorization secret to protect the broadcasting endpoint (see Ruby docs).

--redis_url (ANYCABLE_REDIS_URL or REDIS_URL)

Redis URL for pub/sub (default: "redis://localhost:6379/5").

--redis_channel (ANYCABLE_REDIS_CHANNEL)

Redis channel for broadcasting (default: "__anycable__"). When using the redisx adapter, it's used as a name of the Redis stream.

--nats_servers (ANYCABLE_NATS_SERVERS)

The list of NATS servers to connect to (default: "nats://localhost:4222").

--nats_channel (ANYCABLE_NATS_CHANNEL)

NATS channel for broadcasting (default: "__anycable__").

--log_level (ANYCABLE_LOG_LEVEL)

Logging level (default: "info").

--debug (ANYCABLE_DEBUG)

Enable debug mode (more verbose logging).

--shutdown_timeout (ANYCABLE_SHUTDOWN_TIMEOUT)

The number of seconds to wait for the server to shutdown gracefully, i.e., disconnect all active sessions and perform the corresponding Disconnect RPC calls (see below). Default: 30.

--pong_timeout (ANYCABLE_PONG_TIMEOUT)

For clients using the extended Action Cable protocol, the number of seconds to wait for the client to respond to the PING message with the PONG command. The default value is zero, meaning that no PONGs are expected. The recommended value to activate this feature is 10 seconds. Requiring pongs helps to detect broken connections faster.

Presets

AnyCable-Go comes with a few built-in configuration presets for particular deployments environments, such as Heroku or Fly. The presets are detected and activated automatically. As an indication, you can find a line in the logs:

INFO ... context=config Loaded presets: flyCopy to clipboardErrorCopied

To disable automatic presets activation, provide ANYCABLE_PRESETS=none environment variable (or pass the corresponding option to the CLI: anycable-go --presets=none).

NOTE: Presets do not override explicitly provided configuration values.

Preset: fly

Automatically activated if all of the following environment variables are defined: FLY_APP_NAME, FLY_REGION, FLY_ALLOC_ID.

The preset provide the following defaults:

• host: "0.0.0.0"
• http_broadcast_port: $PORT (set to the same value as the main HTTP port).
• broadcast_adapter: "http" (unless Redis is configured)
• enats_server_addr: "nats://0.0.0.0:4222"
• enats_cluster_addr: "nats://0.0.0.0:5222"
• enats_cluster_name: "<FLY_APP_NAME>-<FLY_REGION>-cluster"
• enats_cluster_routes: "nats://<FLY_REGION>.<FLY_APP_NAME>.internal:5222"
• enats_gateway_advertise: "<FLY_REGION>.<FLY_APP_NAME>.internal:7222" (NOTE: You must set ANYCABLE_ENATS_GATEWAY to nats://0.0.0.0:7222 and configure at least one gateway address manually to enable gateways).

Also, embedded NATS is enabled automatically if no other pub/sub adapter neither Redis is configured. Similarly, pub/sub, broker and broadcast adapters using embedded NATS are configured automatically, too. Thus, by default, AnyCable-Go setups a NATS cluster automatically (within a single region), no configuration is required.

If the ANYCABLE_FLY_RPC_APP_NAME env variable is provided, the following defaults are configured as well:

• rpc_host: "dns:///<FLY_REGION>.<ANYCABLE_FLY_RPC_APP_NAME>.internal:50051"

Preset: heroku

Automatically activated if all of the following environment variables are defined: HEROKU_DYNO_ID, HEROKU_APP_ID. NOTE: These env vars are defined only if the Dyno Metadata feature is enabled.

The preset provides the following defaults:

• host: "0.0.0.0".
• http_broadcast_port: $PORT (to make HTTP endpoint accessible from other applications).

Per-client settings

A client MAY override default values for the settings listed below by providing the corresponding parameters in the WebSocket URL query string:

• ?pi=<seconds>: ping interval (overrides --ping_interval).
• ?ptp=<s | ms | ns>: ping timestamp precision (overrides --ping_timestamp_precision).

For example, using the following URL, you can set the ping interval to 10 seconds and the timestamp precision to milliseconds:

ws://localhost:8080/cable?pi=10&ptp=msCopy to clipboardErrorCopied

HTTP RPC

When using HTTP RPC, you can specify the following additional options:

• http_rpc_secret: a secret token used to authenticate RPC requests.
• http_rpc_timeout: timeout for RPC requests (default: 3s).

You MUST use rpc_host configuration option to provide the URL for HTTP RPC, e.g.: https://my.web.app/anycable.

Please, refer to the RPC over documentation for more information about this communication mode.

TLS

To secure your anycable-go server provide the paths to SSL certificate and private key:

anycable-go --port=443 -ssl_cert=path/to/ssl.cert -ssl_key=path/to/ssl.key

=> INFO time context=http Starting HTTPS server at 0.0.0.0:443Copy to clipboardErrorCopied

If your RPC server requires TLS you can enable it via --rpc_enable_tls (ANYCABLE_RPC_ENABLE_TLS).

If RPC server uses certificate issued by private CA, then you can pass either its file path or PEM contents with --rpc_tls_root_ca (ANYCABLE_RPC_TLS_ROOT_CA).

If RPC uses self-signed certificate, you can disable RPC server certificate verification by setting --rpc_tls_verify (ANYCABLE_RPC_TLS_VERIFY) to false, but this is insecure, use only in test/development.

Concurrency settings

AnyCable-Go uses a single Go gRPC client* to communicate with AnyCable RPC servers (see the corresponding PR). We limit the number of concurrent RPC calls to avoid flooding servers (and getting ResourceExhausted exceptions in response).

* A single client doesn't necessary mean a single connection; a Go gRPC client could maintain multiple HTTP2 connections, for example, when using DNS-based load balancing.

We limit the number of concurrent RPC calls at the application level (to prevent RPC servers overload). By default, the concurrency limit is equal to 28, which is intentionally less than the default RPC size (see Ruby configuration): there is a tiny lag between the times when the response is received by the client and the corresponding worker is returned to the pool. Thus, whenever you update the concurrency settings, make sure that the AnyCable-Go value is slightly less than the AnyCable-RPC one.

You can change this value via --rpc_concurrency (ANYCABLE_RPC_CONCURRENCY) parameter.

Adaptive concurrency

AnyCable-Go Pro provides the adaptive concurrency feature. When it is enabled, AnyCable-Go automatically adjusts its RPC concurrency limit depending on the two factors: the number of ResourceExhausted errors (indicating that the current concurrency limit is greater than RPC servers capacity) and the number of pending RPC calls (indicating the current concurrency is too small to process incoming messages). The first factor (exhausted errors) has a priority (so if we have both a huge backlog and a large number of errors we decrease the concurrency limit).

You can enable the adaptive concurrency by specifying 0 as the --rpc_concurrency value:

$ anycable-go --rpc_concurrency=0

...

INFO 2023-02-23T15:26:13.649Z context=rpc RPC controller initialized: \
  localhost:50051 (concurrency: auto (initial=25, min=5, max=100), enable_tls: false, proto_versions: v1)Copy to clipboardErrorCopied

You should see the (concurrency: auto (...)) in the logs. You can also specify the upper and lower bounds for concurrency via the following parameters:

$ anycable-go \
  --rpc_concurrency=0 \
  --rpc_concurrency_initial=30 \
  --rpc_concurrency_max=50 \
  --rpc_concurrency_min=5Copy to clipboardErrorCopied

You can also monitor the current concurrency value via the rpc_capacity_num metrics.

Disconnect settings

AnyCable-Go notifies an RPC server about disconnected clients asynchronously with a rate limit. We do that to allow other RPC calls to have higher priority (because live clients are usually more important) and to avoid load spikes during mass disconnects (i.e., when a server restarts).

That could lead to the situation when the disconnect queue is overwhelmed, and we cannot perform all the Disconnect calls during server shutdown. Thus, RPC server may not receive all the disconnection events (i.e., disconnect and unsubscribed callbacks in your code).

If you rely on disconnect callbacks in your code, you can tune the default disconnect queue settings to provide better guarantees*:

--disconnect_rate (ANYCABLE_DISCONNECT_RATE)

The max number of Disconnect calls per-second (default: 100).

--disconnect_timeout (ANYCABLE_DISCONNECT_TIMEOUT)

The number of seconds to wait before forcefully shutting down a disconnect queue during the server graceful shutdown (default: 5).

Thus, the default configuration can handle a backlog of up to 500 calls. By increasing both values, you can reduce the number of lost disconnect notifications.

--disconnect_mode (ANYCABLE_DISCONNECT_MODE)

This parameter defines when a Disconnect call should be made for a session. The default is "auto", which means that the Disconnect call is made only if we detected the client interest in it. Currently, we only skip Disconnect calls for sessions authenticated via JWT and using signed streams (Hotwire or CableReady).

Other available modes are "always" and "never". Thus, to disable Disconnect call completely, use --disconnect_mode=never.

Using --disconnect_mode=always is useful when you have some logic in the ApplicationCable::Connetion#disconnect method and you want to invoke it even for JWT and signed streams sessions.

NOTE: AnyCable tries to make a Disconnect call for active sessions during the server shutdown. However, if the server is killed with kill -9 or crashes, the disconnect queue is not flushed, and some disconnect events may be lost. If you experience higher queue sizes during deployments, consider increasing the shutdown timeout by tuning the --shutdown_timeout parameter.

Slow drain mode

AnyCable-Go PRO provides the slow drain mode for disconnecting clients during shutdown. When it is enabled, AnyCable do not try to disconnect all active clients as soon as a shutdown signal is received. Instead, spread the disconnects over the graceful shutdown period. This way, you can reduce the load on AnyCable servers during deployments (i.e., avoid the thundering herd situation).

You can enable this feature by providing the --shutdown_slowdrain option or setting the ANYCABLE_SHUTDOWN_SLOWDRAIN environment variable to true. You should see the following log message on shutdown indicating that the slow drain mode is enabled:

INFO 2023-08-04T07:16:14.339Z context=node Draining 1234 active connections slowly for 24.7sCopy to clipboardErrorCopied

The actual drain period is slightly less than the shutdown timeout—we need to reserve some time to complete RPC calls. Also, there is a maximum interval between disconnects (500ms), so we don't wait too long when the number of clients is not that big.

GOMAXPROCS

We use automaxprocs to automatically set the number of OS threads to match Linux container CPU quota in a virtualized environment, not a number of visible CPUs (which is usually much higher).

This feature is enabled by default. You can opt-out by setting GOMAXPROCS=0 (in this case, the default Go mechanism of defining the number of threads is used).

You can find the actual value for GOMAXPROCS in the starting logs:

INFO 2022-06-30T03:31:21.848Z context=main Starting AnyCable 1.2.0-c4f1c6e (with mruby 1.2.0 (2015-11-17)) (pid: 39705, open file limit: 524288, gomaxprocs: 8)Copy to clipboardErrorCopied