Data-aware scheduling is a feature of the Data Affinity add-on product. Data Affinity reduces network traffic between hosts in the cluster and allows for more flexible scaling of data-intensive applications with improved performance.The data-aware scheduling feature allows Platform Symphony to intelligently schedule application tasks and improve performance by taking into account data location when dispatching the tasks. By directing tasks to resources that already contain the required data, application runtimes can be significantly reduced. As well, this feature can help to meet the challenges of latency requirements for real time applications.
An additional software license is required to use this feature in a grid environment. Use of this feature with Symphony DE does not require a license. This feature is packaged with Symphony and does not require separate deployment.
Follow these steps when you want to add or update a license for the data-aware scheduling feature.
Workload schedulers focus on dispatching tasks to compute hosts and transferring data either directly to the compute hosts or delegating data retrieval to the service. The time required for this data transfer from various sources to where the work is being processed can lead to inefficient use of CPU cycles and underutilization of the resource. With the data-aware scheduling feature, you can specify a preferential association between a task and a service instance or host that already possesses the data required to process the workload. This association is based on the evaluation of a user-defined expression containing data attributes capable of being collected. The evaluation of this expression is carried out against the data attributes of each service instance available to the session. Typically, a data attribute is an identifier for a dataset that is already available to a service instance before it processes workload.
The following example illustrates the concept of data-aware scheduling at the task level. The data preference expression has been evaluated and it is determined that a task in the queue prefers to run on a service instance where the service instance already possesses Dataset1. The SSM collects metadata (service attributes) from all the resources available to the session at that moment. Service B with Dataset1 is currently available and since it is the best match for that task according to the specified preference, the task is subsequently dispatched to Service B.
Attributes take the form of identifiers and can be specified with the +, -, *, and / operators within an expression. An example of an expression would be "DataSet1 + DataSet2", meaning that it is preferred to send workload to a service instance that possesses DataSet1 and DataSet2. The operators can also be used to normalize disparate terms such as data and memory or to give weight to specific terms.
Attribute names have a 32 character limit and can only contain alphanumeric and underscore characters; if you want to use data attributes with names that do not comply with these rules, aliases must be defined. A resource attribute definition can be used to define an alias and to override the default value for an attribute should the session level default or system default be inappropriate.
The result of each expression is a numeric value that is obtained by applying the operators to the attributes in the expression. If the preferred data is available to a service, it should programmatically publish a value for the attribute. Alternatively, the value of the attribute may be collected from the data-aware scheduling plug-in, if present. Once the result is obtained for each resource being evaluated, it is used to sort the resources in ascending order. This means the resource that evaluates to the lowest value is the most preferred.
When no information is available for a resource attribute that is involved in a resource evaluation, the resolution of the expression still proceeds. In such cases, Symphony attempts to substitute a default value for each attribute that it cannot resolve. The value of the attribute is resolved by the system in the following manner, in the given order:
Once a preference is specified on a task within a session, the best match for that task with every service instance currently assigned to the session is considered during the dispatching of the task. This means that once a service becomes available, it is given the next task with the best match for that service instance at the current moment. It is important to note that the next task to be dispatched may not currently be at the front of the queue within the session, i.e., the order of task dispatch depends on the currently available service instances and the preference associated with the tasks and not on the order that tasks are submitted to the session.
Note that the data-aware scheduling feature does not affect the behavior of tasks with the PriorityTask setting enabled. Those tasks are still dispatched before other tasks.
When you want to specify a preference for specific data, the expectation is that attributes in the expression would have their default value set to some number representing the cost of moving this data to that resource.
For example, if we consider the case where a task within a session requires datasets Dataset1, Dataset2 and Dataset3, the preference for the task could be represented as "Dataset1 + Dataset2 + Dataset3". Since expressions are evaluated and sorted in ascending order, we would expect the most preferred resource for this task would have the lowest cost to get the data that the task requires to execute.
Since a service instance is typically able to inform the middleware about what resource attributes (i.e. data sets) it has access to, a fixed value can be substituted for any missing attributes when evaluating the task preference against this service instance. If you use the same fixed value to represent each missing attribute, the system will assign each missing data set the same cost so service instances that have access to the most data sets would be the most preferred and so on. But in reality, it is not generally true that all missing data sets would have the same cost to acquire, especially in the case where the data sets can vary greatly in size or be located on hosts with disparate network topologies and network access speeds. That is why to gain the most benefit from the data-aware scheduling feature, the default value for a data attribute should be carefully considered when applied to a preference expression to ensure that it is representative in some way of the cost of retrieving the missing data for the service instance. This default value is set through the resource attribute definition API.
This direct plug-in provides an additional way for Symphony to obtain the location and cost data associated with preferred resources. The plug-in enhances data-aware scheduling by allowing Symphony to get this data not only from the service side publishing/subscription mechanism, but also from a direct call to a centralized process that can be customized for integration with 3rd party products.
The plug-in process requests the information from an external application and returns it back to the SSM to be used by Symphony as the preferred scheduling location and value for the attribute in the same way as if it were published by a service. Sample code for the plug-in process is provided in an appendix at the end of this document.
To improve performance, this feature also provides caching for the result received from the plug-in so subsequent calls for the same attribute will return cached values.
Here is the sequence the SSM follows to obtain the resource attribute value when data-aware scheduling is enabled.
If the plug-in is configured and the preferred attributes do not have any published values during the current evaluation cycle, the SSM requests locations and values from the plug-in. The following describes the functional flow when the plug-in is used to retrieve resource attribute data for a task.
Upon the task's submission, the SSM parses the preferences and puts the task in the pending queue.
The preferences are evaluated and the SSM tries to find values for the corresponding attributes. Since, in this case, the attribute was not published by the service, the expression is not resolved.
The SSM checks if the direct plug-in is configured for the application. The direct plug-in process is started using a startup command defined in the application profile.
The SSM repeatedly calls the plug-in interface with the attribute names to retrieve the corresponding locations and values.
Tasks are dispatched to currently available resources for the session that are the best match according to the preferred attributes.
Data-aware scheduling is enabled at the application level with the schedulingAffinity attribute in the Consumer element of the application profile. When the attribute is set to DataAware, the SSM collects data attributes of service instances and hosts and evaluates them against a user-defined preference expression. Note that setting the attribute to DataAware automatically enables resource-aware scheduling. When the attribute is set to None (default), no metadata is collected by the SSM and no preference is applied.
The schedulingAffinity attribute can be configured through the PMC or by manually editing the application profile.
Configuring data-aware scheduling for tasks is only possible through the client-side API. For more information about the API, refer to the API reference documentation.
You can define a substitute value for attributes in a preference expression in the event that those attributes are unknown at the time of evaluation. If this default value is not specified, the system default, i.e., 1.0E+300, is substituted.
SessionCreationAttributes.setDefaultResourceAttributeValue (float)
You can programmatically describe data preference for resources.
This method creates an empty preference that can be populated by data attribute definitions.
This method allows you to construct an expression for the data preference.
ResourcePreference ("Dataset1 + Dataset2")
This expression means that it is preferred to run this workload on a service that has Dataset1 and Dataset2.
You are not required to use an attribute definition with your preference unless your application has the following requirements:
The AddDefinition(ResourceAttributeDefinition) method allows a definition to be associated with a data preference.
ResourceAttributeDefinition A ("/dir/filenameA.dat", "A", 100);ResourceAttributeDefinition B ("/dir/filenameB.dat", "B", 150);ResourcePreference rp ();rp.addDefinition(A);rp.addDefinition(B);
This code would result in the following data preference expression: "A + B".
As a best practice, you should call addDefinition() on the SessionCreationAttributes object at session creation time as it is more efficient than calling addDefinition on the ResourcePreference object that is associated with each task.
The ResourceAttributes object allows you to publish details about resource attributes such as datasets that are available to the service instance. The application makes these details available to the middleware to help with optimizing the scheduling in the event the workload defines a preference that includes this attribute. Resource attributes can exist either at a service instance level or be generally available to any service instance on a host.
ResourceAttribute.setName(name)
ResourceAttribute.setValue(value)
ResourceAttribute.setScope(Host)
ServiceContext.publish(ResourceAttribute)
A service can unpublish resource attributes using the ServiceContext.unPublish(ResourceAttribute) method. When calling unPublish(), only the name of the resource attribute is required.
As a best practice, when you publish details about datasets that have been acquired by the service, you can treat the data as having no cost to acquire. This means a value of ‘0’ would be ideal for these types of attributes.