Hardware Locality (hwloc)
1.7.2
|
hwloc is usually used for consulting and manipulating single machine topologies. This includes large systems as long as a single instance of the operating system manages the entire system. However it is sometimes desirable to have multiple independent hosts inside the same topology, for instance when applying algorithms to an entire cluster topology. hwloc therefore offers the ability to agregate multiple host topologies into a single global one.
A multi-node topology contains several single-node topologies. Those are assembled by making their own root objects (usually Machine object) children of higher objects. These higher objects include at least the root of the global topology (usually a System object). Some intermediate objects may also exists, for instance to represent switches in a large fabric.
There are actually three possible types of objects that have different properties with respect to cpusets, nodesets and binding. Indeed those cpusets and nodesets were designed for execution and memory binding within a single operating system. Binding on another system or across several different systems would be meaningless.
It is important to keep this in mind before binding using multi-node topologies. To make sure binding on an object is possible, one should first check that its cpuset or nodeset pointer is not NULL. Then, one should check whether the object is indeed local.
To find out which machine a given object corresponds to, one may look at the info attributes of the parent Machine object. The HostName
info is usually available in Machine objects, it may be retrieved with the following code:
hwloc_obj_t machine_obj; obj = hwloc_get_ancestor_obj_by_type(topology, HWLOC_OBJ_MACHINE, obj); if (machine_obj) return hwloc_obj_get_info_by_name(machine_obj, "HostName"); else return NULL;
The hwloc assembler scripts (see below) also add AssemblerName
and AssemblerIndex
info attributes to the Machine objects to identify the corresponding host name and index during assembly.
One way to manipulate multinode topologies is to retrieve other nodes' topologies as XML files and combine them as a global XML topology. It may then be loaded with hwloc_topology_set_xml() or with the HWLOC_XMLFILE environment variable.
The hwloc-assembler and hwloc-assembler-remote utilities offer the ability to combine XML topologies or remote nodes' topologies (see Command-Line Tools).
The hwloc programming interface offers the ability to build multinode topologies using the custom interface. A new multinode topology has to be initialized with hwloc_topology_init() and then set to custom with hwloc_topology_set_custom(). Topologies and objects mat then be assembled. Later, the custom topology is finalized as usual with hwloc_topology_load().
A custom topology starts with a single root object of type System. It may be modified by inserting a new child object with hwloc_custom_insert_group_object_by_parent() or by duplicating another topology with hwloc_custom_insert_topology(). Both of these operations require to specify the parent object in the custom topology where the insertion will take place. This parent may be either the root (returned by hwloc_get_root_obj()) or an already-inserted object (returned by hwloc_custom_insert_group_object_by_parent()).
Ideally, any existing object in the custom topology could be the parent. However, special care should be taken when traversing the topology to find such an object because most links between objects (children, siblings, cousins) are not setup until hwloc_topology_load() is invoked.
If the topologies of two hosts have been previously gathered in XML files host1.xml
and host2.xml
, the global topology may be assembled with the following code.
hwloc_topology_t host1, host2, global; /* initialize global topology */ hwloc_topology_init(&global); hwloc_topology_set_custom(global); /* insert host1 entire topology below the global topology root */ hwloc_topology_init(&host1); hwloc_topology_load(host1); hwloc_custom_insert_topology(global, hwloc_get_root_obj(global), host1, NULL); hwloc_topology_destroy(host1); /* insert host2 entire topology below the global topology root */ hwloc_topology_init(&host2); hwloc_topology_load(host2); hwloc_custom_insert_topology(global, hwloc_get_root_obj(global), host2, NULL); hwloc_topology_destroy(host2); /* load and play with the global topology */ hwloc_topology_load(global); ...
If a intermediate object such as a switch should be inserted above one of the host topologies:
... /* insert a switch object below the global topology root */ hwloc_obj_t sw = hwloc_custom_insert_group_object_by_parent(global, hwloc_get_root_obj(global), 0); /* insert host2 entire topology below the switch */ hwloc_topology_init(&host2); hwloc_topology_load(host2); hwloc_custom_insert_topology(global, switch, host2, NULL); hwloc_topology_destroy(host2); /* load and play with the global topology */ hwloc_topology_load(global); ...