LATEST VERSION: 8.2.6 - CHANGELOG
Pivotal GemFire® v8.2

Configuring a Multi-site (WAN) System

Configuring a Multi-site (WAN) System

Plan and configure your multi-site topology, and configure the regions that will be shared between systems.

Prerequisites

Before you start, you should understand how to configure membership and communication in peer-to-peer systems using locators. See Configuring Peer-to-Peer Discovery and Configuring Peer Communication. WAN replication is not supported when using multicast for discovery.

WAN deployments increase the messaging demands on a GemFire system. To avoid hangs related to WAN messaging, always set conserve-sockets=false for GemFire members that participate in a WAN deployment. See Configuring Sockets in Multi-Site (WAN) Deployments and Making Sure You Have Enough Sockets.

Note: If you will be upgrading an existing WAN deployment to use the new WAN API, also refer to the WAN deployment upgrade guidelines in the Pivotal GemFire Release Notes.

Main Steps

Use the following steps to configure a multi-site system:

  1. Plan the topology of your multi-site system. See Multi-site (WAN) Topologies for a description of different multi-site topologies.
  2. Configure membership and communication for each distributed system in your multi-site system. You must use locators for peer discovery in a WAN configuration. See Configuring Peer-to-Peer Discovery. Start each distributed system using a unique distributed-system-id and identify remote clusters using remote-locators. For example:
    mcast-port=0
    locators=<locator1-address>[<port1>],<locator2-address>[<port2>]
    distributed-system-id=1
    remote-locators=<remote-locator-addr1>[<port1>],<remote-locator-addr2>[<port2>]
  3. Configure the gateway senders that you will use to distribute region events to remote systems.See Configure Gateway Senders.
  4. Create the data regions that you want to participate in the multi-site system, specifying the gateway sender(s) that each region should use for WAN distribution. Configure the same regions in the target clusters to apply the distributed events. See Create Data Regions for Multi-site Communication.
  5. Configure gateway receivers in each GemFire cluster that will receive region events from another cluster. See Configure Gateway Receivers.
  6. Start distributed system member processes in the correct order (locators first, followed by data nodes) to ensure efficient discovery of WAN resources. See Starting Up Your System.
  7. (Optional.) Deploy custom conflict resolvers to handle resolve potential conflicts that are detected when applying events from over a WAN. See Resolving Conflicting Events.
  8. (Optional.) Deploy WAN filters to determine which events are distributed over the WAN, or to modify events as they are distributed over the WAN. See Filtering Events for Multi-Site (WAN) Distribution.
  9. (Optional.) Configure persistence, conflation, and/or dispatcher threads for gateway sender queues using the instructions in Configuring Multi-Site (WAN) Event Queues.

Configure Gateway Senders

Each gateway sender configuration includes:

  • A unique ID for the gateway sender configuration.
  • The distributed system ID of the remote site to which the sender propagates region events.
  • A property that specifies whether the gateway sender is a serial gateway sender or a parallel gateway sender.
  • Optional properties that configure the gateway sender queue. These queue properties determine features such the amount of memory used by the queue, whether the queue is persisted to disk, and how one or more gateway sender threads dispatch events from the queue.
Note: To configure a gateway sender that uses gfsh to create the cache.xml configurations described below, see create gateway-sender.
See WAN Configuration for more information about individual configuration properties.
  1. For each GemFire system, choose the members that will host a gateway sender configuration and distribute region events to remote sites:
    • You must deploy a parallel gateway sender configuration on each GemFire member that hosts a region that uses the sender.
    • You may choose to deploy a serial gateway sender configuration on one or more GemFire members in order to provide high availability. However, only one instance of a given serial gateway sender configuration distributes region events at any given time.
  2. Configure each gateway sender on a GemFire member using gfsh, cache.xml or Java API:
    • gfsh configuration command
      gfsh>create gateway-sender --id="sender2" --parallel=true --remote-distributed-system-id="2"
      
      gfsh>create gateway-sender --id="sender3" --parallel=true --remote-distributed-system-id="3"
    • cache.xml configuration
      These example cache.xml entries configure two parallel gateway senders to distribute region events to two remote GemFire clusters (clusters "2" and "3"):
      <cache>
        <gateway-sender id="sender2" parallel="true" 
         remote-distributed-system-id="2"/> 
        <gateway-sender id="sender3" parallel="true" 
         remote-distributed-system-id="3"/> 
         ... 
      </cache>
    • Java configuration

      This example code shows how to configure a parallel gateway sender using the GemFire API:

      // Create or obtain the GemFire cache
      Cache cache = new CacheFactory().create();
      
      // Configure and create the gateway sender
      GatewaySenderFactory gateway = cache.createGatewaySenderFactory();
      gateway.setParallel(true);
      GatewaySender sender = gateway.create("sender2", "2");
      sender.start();
      
  3. Depending on your applications, you may need to configure additional features in each gateway sender. Things you need to consider are:
    • The maximum amount of memory each gateway sender queue can use. When the queue exceeds the configured amount of memory, the contents of the queue overflow to disk. For example:
      gfsh>create gateway-sender --id=sender2 --parallel=true --remote-distributed-system-id=2 --maximum-queue-memory=150
      In cache.xml:
      <gateway-sender id="sender2" parallel="true"
       remote-distributed-system-id="2" 
       maximum-queue-memory="150"/> 
    • Whether to enable disk persistence, and whether to use a named disk store for persistence or for overflowing queue events. See Persisting an Event Queue. For example:
      gfsh>create gateway-sender --id=sender2 --parallel=true --remote-distributed-system-id=2 \
      --maximum-queue-memory=150 --enable-persistence=true --disk-store-name=cluster2Store
      In cache.xml:
      <gateway-sender id="sender2" parallel="true"
       remote-distributed-system-id="2" 
       enable-persistence="true" disk-store-name="cluster2Store"
       maximum-queue-memory="150"/> 
    • The number of dispatcher threads to use for processing events from each each gateway queue. The dispatcher-threads attribute of the gateway sender specifies the number of threads that process the queue (default of 5). For example:
      gfsh>create gateway-sender --id=sender2 --parallel=true --remote-distributed-system-id=2 \
      --dispatcher-threads=2 --order-policy=partition
      In cache.xml:
      <gateway-sender id="sender2" parallel="false"
       remote-distributed-system-id="2" 
       dispatcher-threads="2" order-policy="partition"/>
      Note: When multiple dispatcher threads are configured for a serial queue, each thread operates on its own copy of the gateway sender queue. Queue configuration attributes such as maximum-queue-memory are repeated for each dispatcher thread that you configure.

      See Configuring Dispatcher Threads and Order Policy for Event Distribution.

    • For serial gateway senders (parallel=false) that use multiple dispatcher-threads, also configure the ordering policy to use for dispatching the events. See Configuring Dispatcher Threads and Order Policy for Event Distribution.
    • Determine whether you should conflate events in the queue. See Conflating Events in a Queue.
    • Decide whether the gateway sender should start automatically or requires a manual start. This is controlled by the manual-start attribute. For example, in gfsh:
      gfsh>create gateway-sender --id=sender2 --parallel=true --remote-distributed-system-id=2 --manual-start=true
      In cache.xml:
      <gateway-sender id="sender2" parallel="false"
       remote-distributed-system-id="2" 
       dispatcher-threads="2" order-policy="partition" manual-start="true" />
      By default, this is set to false on gateway senders and the senders are started automatically.
    Note: You can configure additional features using parameters of the gfsh create gateway-sender command. See create gateway-sender for more information.
  4. If you have configured the gateway sender with manual-start equal to true, then start the gateway sender when appropriate. In gfsh:
    gfsh> start gateway-sender --id=sender2
Note: The gateway sender configuration for a specific sender id must be identical on each GemFire member that hosts the gateway sender.

Create Data Regions for Multi-site Communication

When you are using a multi-site configuration, you choose which data regions to share between sites. Because of the high cost of distributing data between disparate geographical locations, not all changes are passed between sites. After you define gateway senders, configure regions to use the gateway sender to distribute region events.

  • gfsh Configuration
    gfsh>create region --name=customer --gateway-sender-id=sender2,sender3
    or to modify an existing region:
    gfsh>alter region --name=customer --gateway-sender-id=sender2,sender3
  • cache.xml Configuration
    Use the gateway-sender-ids region attribute to add gateway senders to a region. To assign multiple gateway senders, use a comma-separated list. For example:
    <region-attributes gateway-sender-ids="sender2,sender3">
    </region-attributes>
  • Java API Configuration
    This example shows adding two gateway senders (configured in the earlier example) to a partitioned region:
    RegionFactory rf = 
      cache.createRegionFactory(RegionShortcut.PARTITION);
    rf.addCacheListener(new LoggingCacheListener());
    rf.addGatewaySenderId("sender2");
    rf.addGatewaySenderId("sender3");
    custRegion = rf.create("customer");
    Note: When using the Java API, you must configure a parallel gateway sender before you add its id to a region. This ensures that the sender distributes region events that were persisted before new cache operations take place. If the gateway sender id does not exist when you add it to a region, you receive an IllegalStateException.
Note: Replicated regions cannot use a parallel gateway sender. Use a serial gateway sender instead.
Note: In addition to configuring regions with gateway senders to distribute events, you must configure the same regions in the target clusters to apply the distributed events. The region name in the receiving cluster must exactly match the region name in the sending cluster.

Configure Gateway Receivers

Always configure a gateway receiver in each GemFire cluster that will receive and apply region events from another cluster.

A gateway receiver configuration can be applied to multiple GemFire servers for load balancing and high availability. However, each GemFire member that hosts a gateway receiver must also define all of the regions for which the receiver may receive an event. If a gateway receiver receives an event for a region that the local member does not define, GemFire throws an exception. See Create Data Regions for Multi-site Communication.

Note: You can only host one gateway receiver per member.

A gateway receiver configuration specifies a range of possible port numbers on which to listen. The GemFire server picks an unused port number from the specified range to use for the receiver process. You can use this functionality to easily deploy the same gateway receiver configuration to multiple members.

You can optionally configure gateway receivers to provide a specific IP address or hostname for gateway sender connections. If you configure hostname-for-senders, locators will use the provided hostname or IP address when instructing gateway senders on how to connect to gateway receivers. If you provide "" or null as the value, by default the gateway receiver's bind-address will be sent to clients.

In addition, you can configure gateway receivers to start automatically or to require a manual start. By default, gateway receivers must be started manually.

Note: To configure a gateway receiver, you can use gfsh, cache.xml or Java API configurations as described below. For more information on configuring gateway receivers in gfsh, see create gateway-receiver.
  • gfsh configuration command
    gfsh>create gateway-receiver --start-port=1530 --end-port=1551 
  • cache.xml Configuration

    The following configuration defines a gateway receiver that listens on an unused port in the range from 1530 to 1550:

    <cache>
      <gateway-receiver start-port="1530" end-port="1551" hostname-for-senders="gateway1.mycompany.com" manual-start="false" /> 
       ... 
    </cache>
  • Java Configuration
    // Create or obtain the GemFire cache
    Cache cache = new CacheFactory().create();
    
    // Configure and create the gateway receiver
    GatewayReceiverFactory gateway = cache.createGatewayReceiverFactory();
    gateway.setStartPort(1530);
    gateway.setEndPort(1551);
    gateway.setHostnameForSenders("gateway1.mycompany.com");
    GatewayReceiver receiver = gateway.create();
    receiver.start();
    
    Note: When using the Java API, you must create any region that might receive events from a remote site before you start the gateway receiver. Otherwise, batches of events could arrive from remote sites before the regions for those events have been created. If this occurs, the local site will throw exceptions because the receiving region does not exist yet. If you define regions in cache.xml, the correct startup order is handled automatically.
  • By default, manual-start is configured to true on gateway receivers. If you have not changed this configuration (either in gfsh or cache.xml), you must start your gateway receiver manually. For example, in gfsh:
    gfsh>create gateway-receiver --manual-start=false
  • In gfsh, if you have not disabled manual-start, execute the following command to start the gateway receiver:
    gfsh>start gateway-receiver
  • After starting new gateway receivers, you can execute the load-balance gateway-sender command in gfsh so that a specific gateway sender will be able to rebalance its connections and connect new remote gateway receivers. Invoking this command redistributes gateway sender connections more evenly among all the gateway receivers.
    Note: Another option is to use the GatewaySender.rebalance Java API.
    As an example, let's assume the following scenario:
    1. Start 1 receiver in site ny
    2. Start 4 senders in site ln
    3. Start 3 additional receiver in NY
    You can then execute the following in gfsh to see the effects of rebalancing:
    gfsh -e "connect --locator=localhost[10331]" -e "list gateways"
    ...
    (2) Executing - list gateways
    
    Gateways
    
    GatewaySender
    
    GatewaySender Id |              Member               | Remote Cluster Id |   Type   | Status  | Queued Events | Receiver Location
    ---------------- | --------------------------------- | ----------------- | -------- | ------- | ------------- | -----------------
    ln               | boglesbymac(ny-1:88641)<v2>:33491 | 2                 | Parallel | Running | 0             | boglesbymac:5037
    ln               | boglesbymac(ny-2:88705)<v3>:29329 | 2                 | Parallel | Running | 0             | boglesbymac:5064
    ln               | boglesbymac(ny-3:88715)<v4>:36808 | 2                 | Parallel | Running | 0             | boglesbymac:5132
    ln               | boglesbymac(ny-4:88724)<v5>:52993 | 2                 | Parallel | Running | 0             | boglesbymac:5324
    
    GatewayReceiver
    
                 Member               | Port | Sender Count | Sender's Connected
    --------------------------------- | ---- | ------------ | --------------------------------------------------------------------------
    boglesbymac(ny-1:88641)<v2>:33491 | 5057 | 24           | ["boglesbymac(ln-1:88651)<v2>:48277","boglesbymac(ln-4:88681)<v5>:42784","boglesbymac(ln-2:88662)<v3>:12796","boglesbymac(ln-3:88672)<v4>:43675"]
    boglesbymac(ny-2:88705)<v3>:29329 | 5082 | 0            | []
    boglesbymac(ny-3:88715)<v4>:36808 | 5371 | 0            | []
    boglesbymac(ny-4:88724)<v5>:52993 | 5247 | 0            | []
    
    Execute the load-balance command:
    gfsh -e "connect --locator=localhost[10441]" -e "load-balance gateway-sender --id=ny"...
    
    (2) Executing - load-balance gateway-sender --id=ny
    
                 Member               | Result | Message
    --------------------------------- | ------ |--------------------------------------------------------------------------
    boglesbymac(ln-1:88651)<v2>:48277 | OK     | GatewaySender ny is rebalanced on member boglesbymac(ln-1:88651)<v2>:48277
    boglesbymac(ln-4:88681)<v5>:42784 | OK     | GatewaySender ny is rebalanced on member boglesbymac(ln-4:88681)<v5>:42784
    boglesbymac(ln-3:88672)<v4>:43675 | OK     | GatewaySender ny is rebalanced on member boglesbymac(ln-3:88672)<v4>:43675
    boglesbymac(ln-2:88662)<v3>:12796 | OK     | GatewaySender ny is rebalanced on member boglesbymac(ln-2:88662)<v3>:12796
    Listing gateways in ny again shows the connections are spread much better among the receivers.
    gfsh -e "connect --locator=localhost[10331]" -e "list gateways"...
    
    (2) Executing - list gateways
    
    Gateways
    
    GatewaySender
    
    GatewaySender Id |              Member               | Remote Cluster Id |  Type    | Status  | Queued Events | Receiver Location
    ---------------- | --------------------------------- |  ---------------- | -------- | ------- | ------------- | -----------------
    ln               | boglesbymac(ny-1:88641)<v2>:33491 | 2                 | Parallel | Running | 0             | boglesbymac:5037
    ln               | boglesbymac(ny-2:88705)<v3>:29329 | 2                 | Parallel | Running | 0             | boglesbymac:5064
    ln               | boglesbymac(ny-3:88715)<v4>:36808 | 2                 | Parallel | Running | 0             | boglesbymac:5132
    ln               | boglesbymac(ny-4:88724)<v5>:52993 | 2                 | Parallel | Running | 0             | boglesbymac:5324
    
    GatewayReceiver
    
                 Member               | Port | Sender Count | Sender's Connected
    --------------------------------- | ---- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------
    boglesbymac(ny-1:88641)<v2>:33491 | 5057 | 9            |["boglesbymac(ln-1:88651)<v2>:48277","boglesbymac(ln-4:88681)<v5>:42784","boglesbymac(ln-3:88672)<v4>:43675","boglesbymac(ln-2:88662)<v3>:12796"]
    boglesbymac(ny-2:88705)<v3>:29329 | 5082 | 4            |["boglesbymac(ln-1:88651)<v2>:48277","boglesbymac(ln-4:88681)<v5>:42784","boglesbymac(ln-3:88672)<v4>:43675"]
    boglesbymac(ny-3:88715)<v4>:36808 | 5371 | 4            |["boglesbymac(ln-1:88651)<v2>:48277","boglesbymac(ln-4:88681)<v5>:42784","boglesbymac(ln-3:88672)<v4>:43675"]
    boglesbymac(ny-4:88724)<v5>:52993 | 5247 | 3            |["boglesbymac(ln-1:88651)<v2>:48277","boglesbymac(ln-4:88681)<v5>:42784","boglesbymac(ln-3:88672)<v4>:43675"]
    Running the load balance command in site ln again produces even better balance.
                 Member               | Port | Sender Count | Sender's Connected
    --------------------------------- | ---- | ------------ |-------------------------------------------------------------------------------------------------------------------------------------------------
    boglesbymac(ny-1:88641)<v2>:33491 | 5057 | 7            |["boglesbymac(ln-1:88651)<v2>:48277","boglesbymac(ln-4:88681)<v5>:42784","boglesbymac(ln-2:88662)<v3>:12796","boglesbymac(ln-3:88672)<v4>:43675"]
    boglesbymac(ny-2:88705)<v3>:29329 | 5082 | 3            |["boglesbymac(ln-1:88651)<v2>:48277","boglesbymac(ln-3:88672)<v4>:43675","boglesbymac(ln-2:88662)<v3>:12796"]
    boglesbymac(ny-3:88715)<v4>:36808 | 5371 | 5            |["boglesbymac(ln-1:88651)<v2>:48277","boglesbymac(ln-4:88681)<v5>:42784","boglesbymac(ln-2:88662)<v3>:12796","boglesbymac(ln-3:88672)<v4>:43675"]
    boglesbymac(ny-4:88724)<v5>:52993 | 5247 | 6            |["boglesbymac(ln-1:88651)<v2>:48277","boglesbymac(ln-4:88681)<v5>:42784","boglesbymac(ln-2:88662)<v3>:12796","boglesbymac(ln-3:88672)<v4>:43675"]