Sync Rules - Limit data synced to devices

Sync rules give developers the ability to determine programmatically what data should sync to which devices. Sync rules are important in production applications, where devices should only store data that's needed for the app to work for security and performance reasons.

Sync Rules v1 and v2

The below document details sync rules in apps that have data rules enabled.

• For documentation on sync rules v2 (sync rules without data ACLs), see the legacy docs here. Note that there are no syntax or functional updates for sync rules in apps that have data rules enabled.

• For documentation on the deprecated sync rules v1, go here.

Sync rules vs ACLs

Sync rules specify which data syncs to users' devices, whereas ACLs specify users' access to data objects. This document describes sync rules. You can learn more about ACLs here.

Sync rule reprocessing

When changes to sync rules are deployed, all data for that deployment needs to be processed again to take the new sync rules into consideration. Please read more information about sync reprocessing below.

Data Buckets

Sync rules are defined in terms of data buckets. It is important that you are familiar with these before moving on to the next sections. Read up about them here:

Data Buckets

Data is synched to devices according to the data buckets defined in the data rules.

Important: If access is limited with data ACLs, syncing could be disabled for a bucket.

See and example of this below.

Examples

Synchronize all data

During development, it is useful to not have to worry about sync rules, and synchronize everything. This is not recommended for applications in production, unless the app will only ever contain a small number of objects (around 10 000 or less).

data-rules.xml

<data-rules version="3">
    <global-bucket>
        <all-models />
    </global-bucket>
</data-rules>

Synchronize all data for specific users

It is also possible to synchronize all data only to specific users only, by using the via attribute.

data-rules.xml

<data-rules version="3">
    <global-bucket via="self[developer == true]">
        <!-- All data for developers -->
        <all-models />
    </global-bucket>
    
    <bucket via="self[developer == false]/region">
        <!-- Region-specific data for other users -->
        <has-many name="clients" />
    <bucket>
</data-rules>

Synchronize only a subset of data within a model

Here we specify a condition to only synchronize a subset of the objects within the category model.

data-rules.xml

<data-rules version="3">
    <global-bucket>
        <!-- Sync only non-archived category objects -->
        <model name="category" condition="archived != true" />
    </global-bucket>
</data-rules>

Synchronize data for certain users only

Here we specify the via attribute to synchronize all clients in a user's region.

data-rules.xml

<data-rules version="3">
    <!-- 'via' specifies how we get from the user to the region -->
    <bucket via="self/region">
        <!-- Sync all clients in the region according to the has-many relationship -->
        <has-many name="clients" />
    </bucket>
</data-rules>

Disable synchronization with the read="online" ACL

data-rules.xml

<data-rules version="3">
    <!-- Neither the region nor its clients are synced -->
    <bucket via="self/region" read="online">
        <has-many name="clients" />
    </bucket>
</data-rules>

Advanced Example

This example combines many of the available configuration options to showcase more complex sync rules:

schema.xml

<?xml version="1.0" encoding="UTF-8"?>
<data-model>
    <model name="user" label="User">
        <field name="name" label="Name" type="text:name"/>
        <field name="role" label="Role" type="single-choice">
            <option key="admin">Admin</option>
            <option key="field_engineer">Field Engineer</option>
        </field>

        <belongs-to model="site" />
        <has-many model="message" inverse-of="sent_by" name="sent_messages" />
        <has-many model="message" inverse-of="received_by" name="received_messages" />

        <display>{name}</display>
    </model>

    <model name="site" label="Site">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="name" label="Name" type="text:name" />

        <has-many model="building" name="buildings" />
        <has-many model="room" name="rooms" />
        <has-many model="equipment" name="equipment" />
        
        <display>{name}</display>
    </model>

    <model name="building" label="Building">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="name" label="Name" type="text:name" />

        <belongs-to model="site" />
        
        <display>{name}</display>
    </model>

    <model name="room" label="Room">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="name" label="Name" type="text:name" />

        <belongs-to model="site" />
        <belongs-to model="building" />
        
        <display>{name}</display>
    </model>

    <model name="category" label="Category">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="name" label="Name" type="text:name" />
        
        <display>{name}</display>
    </model>

    <model name="tool_type" label="Tool Type">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="name" label="Name" type="text:name" />
        
        <display>{name}</display>
    </model>

    <model name="equipment" label="Equipment">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="in_use" label="In Use?" type="boolean" />
        <field name="name" label="Name" type="text:name" />

        <belongs-to model="site" />
        <belongs-to model="room" />
        <belongs-to model="category" />
        <belongs-to model="tool_type" />
        
        <display>{name}</display>
    </model>

    <model name="message" label="Message">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="message" label="Message" type="text" />
        <field name="sent_at" label="Sent At" type="datetime" />
        <field name="received_at" label="Received At" type="datetime" />
        
        <belongs-to model="user" name="sent_by" />
        <belongs-to model="user" name="received_by" />
        <display></display>
    </model>
</data-model>

data_rules.xml

<?xml version="1.0" encoding="UTF-8"?>
<data-rules version="3">
    <!-- Sync the user object itself, required to access 'user' in the app. -->
    <bucket via="self">
        <!-- Also sync all messages that the user has sent or received. -->
        <has-many name="sent_messages" condition="archived != true" />
        <has-many name="received_messages" condition="archived != true" />
    </bucket>
  
    <!-- For non-admin users, sync specific data related to sites. -->
    <bucket via="self[role != 'admin']/site[archived != true]">
        <has-many name="buildings" />
        <has-many name="rooms" />
        <has-many name="equipment" condition="in_use == true" />
    </bucket>
  
    <!-- For admin users, sync data for all sites. -->
    <global-bucket via="self[role == 'admin']">
        <model name="building" />
        <model name="room" />
        <model name="equipment" />
    </global-bucket>
  
    <!-- Some global data is synchronized to all users. -->
    <global-bucket>
        <model name="category" />
        <model name="tool_type" condition="archived != true" />
    </global-bucket>
</data-rules>

Sync Rule Reprocessing

When changes to the sync rules are deployed, all data for that deployment needs to be processed again to take the new sync rules into consideration. The exact processing time varies based on the app and the server load, but it usually takes around 1-2 seconds per 1000 entries in the Oplog, or half an hour per million entries.

Changes that trigger sync rule reprocessing (when deployed) include:

• Changes to models, relationships, and conditions in the sync rules.

• Some (rare) changes in the data model, if it changes the meaning of the sync rules. For example, if the model referenced by a relationship used in sync rules changes, reprocessing will be triggered.

• White-space changes and comments do not trigger sync rule reprocessing.

OXIDE will present a warning dialog when you deploy changes to a Staging / Production deployment that will trigger sync rule reprocessing.

During sync rule reprocessing, users can proceed to use their apps as usual, however previous sync rules will apply until reprocessing is complete.

This means that in cases where the new version of the code relies on new sync rules being active immediately, you should plan on deploying your code in multiple phases:

1. Deploy a change containing only the new sync rules, as well as any new models and fields in the data model.

2. Once sync rule reprocessing is complete, deploy the rest of the changes.

Sync reprocessing status

The reprocessing status for a deployment (including an estimate of the number of Oplog entries to be processed and the reprocessing duration) can be seen on the "Sync Diagnostics" page in the data browser. You may need to elevate permissions to access this page.

Once sync rules are reprocessing, the status can also be seen in OXIDE's Deployment workspace for the corresponding deployment:

Developer Notes

Here are a few final notes on sync rules for developers:

• Once sync rules have been deployed for an app, it will become critically important to update the rules when new models are added to the data model, or if model or relationship names change. Failing to do so will result in queries not showing the necessary data in the view, or logic in the JS failing.

• Sync rules affect what data can be accessed in the app via DB. OnlineDB is unaffected by sync rules.

  • Note: this behavior is different for data ACLs. Please see this section for more details.
• Sync rules constraints have a big impact on data model design. It is often useful to create "sync relationships" which are only used for sync rules, but have no other meaning in the data model.

Guidance around using sync rules

A general rule is to sync only the data that is applicable to a specific app. The fewer data synced, the more performant the app will be in general:

• Fewer than 10k objects on a device is ideal.

• Up to 50k objects synced to a device should work but needs to be tested properly. At these data volumes, the app becomes sensitive to poorly-written and unindexed queries. Be sure to monitor the DB size for increases (in the app diagnostics) to prevent unforeseen performance degradations.

• Up to 500k objects synced to a device can work, but even with performant queries the sync performance will become slow. This will be exacerbated if there are a large number of users syncing the same data or if the data changes often. Strongly consider using aggressive archiving strategies, or OnlineDB.