With the latest and accelerated advances in machine learning (ML), machines can perceive pure language, interact in conversations, draw photographs, create movies and extra. Modern ML fashions are programmed and educated utilizing ML programming frameworks, corresponding to TensorMove, JAX, PyTorch, amongst many others. These libraries present high-level directions to ML practitioners, corresponding to linear algebra operations (e.g., matrix multiplication, convolution, and many others.) and neural community layers (e.g., 2D convolution layers, transformer layers). Importantly, practitioners needn’t fear about the best way to make their fashions run effectively on {hardware} as a result of an ML framework will robotically optimize the consumer’s mannequin by an underlying compiler. The effectivity of the ML workload, thus, will depend on how good the compiler is. A compiler sometimes depends on heuristics to resolve complicated optimization issues, typically ensuing in suboptimal efficiency.
In this weblog publish, we current thrilling developments in ML for ML. In specific, we present how we use ML to enhance effectivity of ML workloads! Prior works, each inner and exterior, have proven that we are able to use ML to enhance efficiency of ML packages by choosing higher ML compiler choices. Although there exist just a few datasets for program efficiency prediction, they aim small sub-programs, corresponding to primary blocks or kernels. We introduce “TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs” (introduced at NeurIPS 2023), which we not too long ago launched to gas extra analysis in ML for program optimization. We hosted a Kaggle competitors on the dataset, which not too long ago accomplished with 792 individuals on 616 groups from 66 nations. Furthermore, in “Learning Large Graph Property Prediction via Graph Segment Training”, we cowl a novel technique to scale graph neural community (GNN) coaching to deal with giant packages represented as graphs. The approach each permits coaching arbitrarily giant graphs on a tool with restricted reminiscence capability and improves generalization of the mannequin.
ML compilers
ML compilers are software program routines that convert user-written packages (right here, mathematical directions offered by libraries corresponding to TensorMove) to executables (directions to execute on the precise {hardware}). An ML program may be represented as a computation graph, the place a node represents a tensor operation (corresponding to matrix multiplication), and an edge represents a tensor flowing from one node to a different. ML compilers have to resolve many complicated optimization issues, together with graph-level and kernel-level optimizations. A graph-level optimization requires the context of the complete graph to make optimum choices and transforms the complete graph accordingly. A kernel-level optimization transforms one kernel (a fused subgraph) at a time, independently of different kernels.
 |
| Important optimizations in ML compilers embrace graph-level and kernel-level optimizations. |
To present a concrete instance, think about a matrix (2D tensor):
 |
It may be saved in laptop reminiscence as [A B C a b c] or [A a B b C c], often known as row- and column-major reminiscence structure, respectively. One necessary ML compiler optimization is to assign reminiscence layouts to all intermediate tensors in this system. The determine beneath reveals two totally different structure configurations for the identical program. Let’s assume that on the left-hand aspect, the assigned layouts (in pink) are probably the most environment friendly choice for every particular person operator. However, this structure configuration requires the compiler to insert a copy operation to rework the reminiscence structure between the add and convolution operations. On the opposite hand, the right-hand aspect configuration is likely to be much less environment friendly for every particular person operator, however it doesn’t require the extra reminiscence transformation. The structure project optimization has to commerce off between native computation effectivity and structure transformation overhead.
 |
| A node represents a tensor operator, annotated with its output tensor form [n0, n1, …], the place ni is the dimensions of dimension i. Layout {d0, d1, …} represents minor-to-major ordering in reminiscence. Applied configurations are highlighted in pink, and different legitimate configurations are highlighted in blue. A structure configuration specifies the layouts of inputs and outputs of influential operators (i.e., convolution and reshape). A replica operator is inserted when there’s a structure mismatch. |
If the compiler makes optimum selections, vital speedups may be made. For instance, we’ve got seen as much as a 32% speedup when selecting an optimum structure configuration over the default compiler’s configuration in the XLA benchmark suite.
TpuGraphs dataset
Given the above, we goal to enhance ML mannequin effectivity by enhancing the ML compiler. Specifically, it may be very efficient to equip the compiler with a discovered value mannequin that takes in an enter program and compiler configuration after which outputs the expected runtime of this system.
With this motivation, we launch TpuGraphs, a dataset for learning value fashions for packages operating on Google’s customized Tensor Processing Units (TPUs). The dataset targets two XLA compiler configurations: structure (generalization of row- and column-major ordering, from matrices, to increased dimension tensors) and tiling (configurations of tile sizes). We present obtain directions and starter code on the TpuGraphs GitHub. Each instance in the dataset incorporates a computational graph of an ML workload, a compilation configuration, and the execution time of the graph when compiled with the configuration. The graphs in the dataset are collected from open-source ML packages, that includes fashionable mannequin architectures, e.g., ResNet, EfficientNet, Mask R-CNN, and Transformer. The dataset supplies 25× extra graphs than the biggest (earlier) graph property prediction dataset (with comparable graph sizes), and graph dimension is 770× bigger on common in comparison with current efficiency prediction datasets on ML packages. With this tremendously expanded scale, for the primary time we are able to discover the graph-level prediction activity on giant graphs, which is topic to challenges corresponding to scalability, coaching effectivity, and mannequin high quality.
 |
| Scale of TpuGraphs in comparison with different graph property prediction datasets. |
We present baseline discovered value fashions with our dataset (structure proven beneath). Our baseline fashions are based mostly on a GNN because the enter program is represented as a graph. Node options, proven in blue beneath, encompass two elements. The first half is an opcode id, crucial data of a node, which signifies the kind of tensor operation. Our baseline fashions, thus, map an opcode id to an opcode embedding through an embedding lookup desk. The opcode embedding is then concatenated with the second half, the remainder of the node options, as inputs to a GNN. We mix the node embeddings produced by the GNN to create the fixed-size embedding of the graph utilizing a easy graph pooling discount (i.e., sum and imply). The ensuing graph embedding is then linearly remodeled into the ultimate scalar output by a feedforward layer.
 |
| Our baseline discovered value mannequin employs a GNN since packages may be naturally represented as graphs. |
Furthermore we current Graph Segment Training (GST), a way for scaling GNN coaching to deal with giant graphs on a tool with restricted reminiscence capability in instances the place the prediction activity is on the entire-graph (i.e., graph-level prediction). Unlike scaling coaching for node- or edge-level prediction, scaling for graph-level prediction is understudied however essential to our area, as computation graphs can comprise tons of of hundreds of nodes. In a typical GNN coaching (“Full Graph Training”, on the left beneath), a GNN mannequin is educated utilizing a whole graph, that means all nodes and edges of the graph are used to compute gradients. For giant graphs, this is likely to be computationally infeasible. In GST, every giant graph is partitioned into smaller segments, and a random subset of segments is chosen to replace the mannequin; embeddings for the remaining segments are produced with out saving their intermediate activations (to keep away from consuming reminiscence). The embeddings of all segments are then mixed to generate an embedding for the unique giant graph, which is then used for prediction. In addition, we introduce the historic embedding desk to effectively receive graph segments’ embeddings and phase dropout to mitigate the staleness from historic embeddings. Together, our full technique quickens the end-to-end coaching time by 3×.
 |
| Comparing Full Graph Training (typical technique) vs Graph Segment Training (our proposed technique). |
Kaggle competitors
Finally, we ran the “Fast or Slow? Predict AI Model Runtime” competitors over the TpuGraph dataset. This competitors ended with 792 individuals on 616 groups. We had 10507 submissions from 66 nations. For 153 customers (together with 47 in the highest 100), this was their first competitors. We discovered many fascinating new strategies employed by the taking part groups, corresponding to:
- Graph pruning / compression: Instead of utilizing the GST technique, many groups experimented with other ways to compress giant graphs (e.g., conserving solely subgraphs that embrace the configurable nodes and their quick neighbors).
- Feature padding worth: Some groups noticed that the default padding worth of 0 is problematic as a result of 0 clashes with a sound characteristic worth, so utilizing a padding worth of -1 can enhance the mannequin accuracy considerably.
- Node options: Some groups noticed that further node options (corresponding to dot common’s contracting dimensions) are necessary. A number of groups discovered that totally different encodings of node options additionally matter.
- Cross-configuration consideration: A successful staff designed a easy layer that enables the mannequin to explicitly “examine” configs towards one another. This approach is proven to be a lot better than letting the mannequin infer for every config individually.
We will debrief the competitors and preview the successful options on the competitors session on the ML for Systems workshop at NeurIPS on December 16, 2023. Finally, congratulations to all of the winners and thanks for your contributions to advancing analysis in ML for programs!
NeurIPS expo
If you have an interest in extra analysis about structured information and synthetic intelligence, we hosted the NeurIPS Expo panel Graph Learning Meets Artificial Intelligence on December 9, which coated advancing discovered value fashions and extra!
Acknowledgements
Sami Abu-el-Haija (Google Research) contributed considerably to this work and write-up. The analysis in this publish describes joint work with many further collaborators together with Mike Burrows, Kaidi Cao, Bahare Fatemi, Jure Leskovec, Charith Mendis, Dustin Zelle, and Yanqi Zhou.