Sync Rules v2 (legacy docs)

Data Rules vs Sync Rules v2

Data rules are a combination sync rules and Access Control Lists, specified in a single XML document. If your app is currently using sync rules v2 (i.e. where sync rules are specified in a sync_rules.xml file), we strongly recommend migrating to data rules. Please see this guide which outlines the migration process.

Details matter with sync rules

Sync rules is an advanced feature where implementation details can have a significant impact on app performance, sync performance, as well as the design of your data model.

Please be sure to understand the constraints and implications of your sync rules before deploying them to an environment with active users.

Sync rule reprocessing

When changes to the sync rules are deployed to the backend, all data for that deployment needs to be processed again to take the new sync rules into consideration. Please read more information about sync reprocessing below.

Sync Rules v1

The below details sync rules v2. For documentation on sync rules v1, please read the legacy docs.

Sync rules give developers the ability to determine programmatically what data should be sync to which devices. Sync rules are important in production applications, where devices should only store data that's needed for the app to work for security and performance reasons.

Developers can specify sync rules for their apps by opening the sync_rules.xml file in OXIDE.

Buckets

Sync rules are defined in terms of buckets. A bucket is used to group together objects to synchronize to one or more users. The goal is to split data into separate buckets to reduce the amount of data synchronized to each user, but to not have so many buckets that sync performance is impacted.

Global buckets

The simplest buckets are global buckets. These buckets are synchronized to all users.

This is typically used for data that stays fairly constant over time, e.g. categories used in drop-down lists.

<sync version="2">
    <global-bucket>
        <model name="category" />
        <model name="subcategory" />
    </global-bucket>
</sync>

Models can also have conditions to only synchronize a subset of them:

    <global-bucket>
        <model name="category" condition="archived != true" />
    </global-bucket>

In some cases, data should only be synchronized to some users, e.g. only to admin users or only to technicians. This is possible by specifying a via property on the bucket.

    <global-bucket via="self[role == 'technician']">
        <model name="part_types" />
    </global-bucket>

More details on the via attribute will follow later.

Object-specific buckets

Object-specific buckets each have a bucket root which defines the grouping. All objects inside the group must have a direct belongs-to relationship to the root object.

Users are then linked to the bucket roots by a path that traverses one or more relationships (belongs-to or has-many).

As an example suppose that we want to split data by region. Each user belongs to a specific region, and we only want to sync client data to users within the same region.

In this case, we'd define the Region as the root of the bucket. We place clients in the bucket according to the belongs-to relationship, and sync the region bucket to each user.

<sync version="2">
    <!-- 'via' specifies how we get from the user to the region -->
    <bucket via="self/region">
        <!-- The root itself (region in this case) is automatically included in the bucket -->
        <!-- sync all clients in the region according to the has-many relationship -->
        <has-many name="clients" />
    </bucket>
</sync>

Suppose clients have additional info that we need to include, e.g. contacts. To include this in the region bucket, it needs an explicit belongs-to relationship to the region.

Here's the updated data model:

<data-model>
    <!-- only relevant fields are listed here -->
    <model name="region">
        <has-many model="user" name="users" />
        <has-many model="client" name="clients" /> 

        <!-- new relationship for contacts -->
        <has-many model="contact" name="contacts" />
    </model>
    <model name="client">
        <belongs-to model="region" />
        <has-many model="contact" name="contacts" />
    </model>
    <model name="contact">
        <belongs-to model="client" />
      
        <!-- New explicit relationship between contact and region.
             Make sure to update this whenever the client changes. -->
        <belongs-to model="region" />
    </model>
</data-model>

And the corresponding Sync Rules:

    <bucket via="self/region">
        <has-many name="clients" />
        <has-many name="contacts" />
    </bucket>

Same as with global-buckets, we can add conditions on the models:

    <bucket via="self/region">
        <has-many name="clients" />
        <has-many name="contacts" condition="number != null"/>
    </bucket>

We can also add conditions on the root object:

    <bucket via="self/region[archived != true]">
          <!-- ... -->
    </bucket>

Or on the user:

    <bucket via="self[role == 'manager']/region[archived != true]">
          <!-- ... -->
    </bucket>

Using via: in depth

In the case of object-specific buckets, the via attribute indicates the path to take from the user object to the root of the bucket. For each user, this path can result in zero, one or many bucket roots.

The via path can contain any number of belongs-to or has-many relationships, along with an optional condition as a filter on each.

Here are some examples:

Path	Description
via="self"	The user object is the bucket root.
via="self[role == 'admin']"	The user object is the bucket root. The bucket is only synchronized to users that have a role of admin.
via="self/region"	The region is the bucket root. Each user can have zero or one region.
via="self/region/clients"	The client is the bucket root. There can be many bucket roots per user, depending on the number of clients.

Limitation: Number of bucket roots

The number of bucket roots per user may never exceed more than 200.

If it does, the user will not be able to synchronize. Sync performance also degrades as the number of buckets per user increases, so it is recommended to keep this to fewer than 50 per user.

Frequently Changing Bucket Roots - pre v4.24

Older versions of the container (before version 4.24) do not handle changing bucket roots per user efficiently. For this reason, it is recommended to keep the bucket roots mostly static per user. For example, in our earlier example, it is not expected that a user's region will change often. If however, we use something like current_assignment as the root, the sync performance will suffer every time current_assignment changes.

Newer container versions (4.24 and later) handle this more efficiently and do not have this issue.

For global buckets, via works similarly. However, instead of having separate bucket roots, there is only a single bucket that will be synchronized if the via field has any results.

Here are some examples:

Path	Description
no via	The bucket is synchronized to all users.
via="self"	Same as above.
via="self[role == 'admin']"	The bucket is only synchronized to users that have a role of admin.
via="self/settings[role == 'admin']"	Same as above, but where the role field is stored on a separate settings object.
via="self/assignments[active == true]"	The bucket is synchronized if the user has one or more assignments that is marked as active.

Synchronizing All Data

During development, it is useful to not have to worry about sync rules, and synchronize everything. This is not recommended for applications in production, unless the app will only ever contain a small number of objects (around 10 000 or less).

<sync version="2">
    <global-bucket>
        <all-models />
    </global-bucket>
</sync>

It is also possible to synchronize all data only to specific users, by using the via field.

<sync version="2">
    <global-bucket via="self[developer == true]">
        <!-- All data for developers -->
        <all-models />
    </global-bucket>
    
    <bucket via="self[developer == false]/region">
        <!-- Region-specific data for other users -->
    <bucket>
</sync>

Conditions

Conditions can be applied to any single field on an object.

The following operators are available:

==
!=
lt or <
lte or <=
gt or >
gte or >=

The following are valid values to compare against:

true, e.g. "archived == true"
false, e.g. "archived != false"
null, e.g. "region != null"
Numbers (including for single-choice-integer), e.g. "count lte 5"
Text (including for single-choice), e.g. "status != 'completed'"

Advanced Example

schema.xml

<?xml version="1.0" encoding="UTF-8"?>
<data-model>
    <model name="user" label="User">
        <field name="name" label="Name" type="text:name"/>
        <field name="role" label="Role" type="single-choice">
            <option key="admin">Admin</option>
            <option key="field_engineer">Field Engineer</option>
        </field>

        <belongs-to model="site" />
        <has-many model="message" inverse-of="sent_by" name="sent_messages" />
        <has-many model="message" inverse-of="received_by" name="received_messages" />

        <display>{name}</display>
    </model>

    <model name="site" label="Site">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="name" label="Name" type="text:name" />

        <has-many model="building" name="buildings" />
        <has-many model="room" name="rooms" />
        <has-many model="equipment" name="equipment" />
        
        <display>{name}</display>
    </model>

    <model name="building" label="Building">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="name" label="Name" type="text:name" />

        <belongs-to model="site" />
        
        <display>{name}</display>
    </model>

    <model name="room" label="Room">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="name" label="Name" type="text:name" />

        <belongs-to model="site" />
        <belongs-to model="building" />
        
        <display>{name}</display>
    </model>

    <model name="category" label="Category">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="name" label="Name" type="text:name" />
        
        <display>{name}</display>
    </model>

    <model name="tool_type" label="Tool Type">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="name" label="Name" type="text:name" />
        
        <display>{name}</display>
    </model>

    <model name="equipment" label="Equipment">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="in_use" label="In Use?" type="boolean" />
        <field name="name" label="Name" type="text:name" />

        <belongs-to model="site" />
        <belongs-to model="room" />
        <belongs-to model="category" />
        <belongs-to model="tool_type" />
        
        <display>{name}</display>
    </model>

    <model name="message" label="Message">
        <field name="archived" label="Archived?" type="boolean" />
        <field name="message" label="Message" type="text" />
        <field name="sent_at" label="Sent At" type="datetime" />
        <field name="received_at" label="Received At" type="datetime" />
        
        <belongs-to model="user" name="sent_by" />
        <belongs-to model="user" name="received_by" />
        <display></display>
    </model>
</data-model>

sync_rules.xml

<?xml version="1.0" encoding="UTF-8"?>
<sync version="2">
    <!-- Sync the user object itself, required to access 'user' in the app. -->
    <bucket via="self">
        <!-- Also sync all messages that the user has sent or received. -->
        <has-many name="sent_messages" condition="archived != true" />
        <has-many name="received_messages" condition="archived != true" />
    </bucket>
  
    <!-- For non-admin users, sync specific data related to sites. -->
    <bucket via="self[role != 'admin']/site[archived != true]">
        <has-many name="buildings" />
        <has-many name="rooms" />
        <has-many name="equipment" condition="in_use == true" />
    </bucket>
  
    <!-- For admin users, sync data for all sites. -->
    <global-bucket via="self[role == 'admin']">
        <model name="building" />
        <model name="room" />
        <model name="equipment" />
    </global-bucket>
  
    <!-- Some global data is synchronized to all users. -->
    <global-bucket>
        <model name="category" />
        <model name="tool_type" condition="archived != true" />
    </global-bucket>
</sync>

Sync Rule Reprocessing

When changes to the sync rules are deployed to the backend, all data for that deployment needs to be processed again to take the new sync rules into consideration. The exact processing time varies based on the app and the server load, but it usually takes around 1-2 seconds per 1000 entries in the Oplog, or half an hour per million entries.

Changes that trigger sync rule reprocessing (when deployed) include:

• Changes to models, relationships, and conditions in the sync rules.

• Some (rare) changes in the data model, if it changes the meaning of the sync rules. For example, if the model referenced by a relationship used in sync rules changes, reprocessing will be triggered.

• White-space changes and comments do not trigger sync rule reprocessing.

OXIDE will present a warning dialog when you deploy changes to a Staging / Production deployment that will trigger sync rule reprocessing.

During sync rule reprocessing, users can proceed to use their apps as usual, however previous sync rules will apply until reprocessing is complete.

This means that in cases where the new version of the code relies on new sync rules being active immediately, you should plan on deploying your code in multiple phases:

1. Deploy a change containing only the new sync rules, as well as any new models and fields in the data model.

2. Once sync rule reprocessing is complete, deploy the rest of the changes.

Tip: The reprocessing status for a deployment (including the number of Oplog entries to be processed) can be seen on the "Sync Diagnostics" page in the data browser. You may need to elevate permissions to access this page.

Developer Notes

Here are a few final notes on sync rules for developers:

• Once sync rules have been deployed for an app, it will become critically important to update the rules when new models are added to the data model, or if model or relationship names change. Failing to do so will result in queries not showing the necessary data in the view, or logic in the JS failing.

• Sync rules affect what data can be accessed in the app via DB. OnlineDB is unaffected by sync rules.

  • Note: this behavior is different for data rules. Please see this section for more details.

• Sync rules constraints have a big impact on data model design. It is often useful to create "sync relationships" which are only used for sync rules, but have no other meaning in the data model.

Guidance around using sync rules

A general rule is to sync only the data that is applicable to a specific app. The fewer data synced, the more performant the app will be in general:

• Fewer than 10k objects on a device is ideal.

• Up to 50k objects synced to a device should work but needs to be tested properly. At these data volumes, the app becomes sensitive to poorly-written and unindexed queries. Be sure to monitor the DB size for increases (in the app diagnostics) to prevent unforeseen performance degradations.

• Up to 500k objects synced to a device can work, but even with performant queries the sync performance will become slow. This will be exacerbated if there are a large number of users syncing the same data or if the data changes often. Strongly consider using aggressive archiving strategies, or OnlineDB.