The best tools for TensorFlow

TensorFlow becomes the de facto standard for creating machine learning models. Find out which tools you can use to get the most out of the framework.

TensorFlow is an open source machine learning framework. It is basically a library for numerical calculations based on data stream graphs. The graph nodes represent mathematical operations, while the edges of the graph represent multidimensional data arrays (tensors) flowing between them.

Tools in TensorFlow

This flexible architecture makes it possible to provide calculations on one or more CPUs or GPUs in a desktop PC, server or mobile device without having to rewrite code. TensorFlow also includes TensorBoard , a toolkit for data visualization .

While TensorFlow built data flow graphs and later executed in sessions. Version 2.0 fully supports the eager execution mode . It is an imperative define-by-run interface that instantly calculates operations without building a graph. The Eager Execution mode supports automatic differentiation via the GradientTape API. One of the improvements in tf.keras (see Keras below) is support for Eager Execution.

TensorFlow 2.0 knock out some other APIs. For existing code (A conversion tool has been built-in, in addition to a compatibility library.

Estimators are the most scalable and production-oriented model types for machine learning from TensorFlow. Businesses can either use Google’s ready- made estimators or write their own. Estimators built on tf.keras.layers. It simplifies customization . It is usually more convenient to build models with estimators rather than using TensorFlow’s low-level APIs. Pre-built estimators make it possible to work at a higher conceptual level than with the basic TensorFlow APIs

Keras

TensorFlow still supports its original low-level API. However, tf.keras is now the preferred high-level API , an implementation of the Keras API standard that contains improvements specific to TensorFlow. “High” and “Low” refer to how deep and “hardware-related” the API applies. Low level means that more detailed but also more complex settings can be made. The functions are abstracted at the high level, so that although fewer options are available, the API is easier to use.

Keras is a high-level API for neural networks. Python is its source code langue . It can run on top of TensorFlow , CNTK , or Theano . Third parties Additionaly supports back ends such as MXNet or PlaidML .

User friendliness in mind is behind Keras development . It should be modular, easy to extend, and written in Python. The API developed for “people and not machines”. It follows best practices. The Intention is to reduce the cognitive load during operation.

Neural layers, cost functions, optimizer, initialization schemes, activation functions and regulation schemes. These are all independent modules in Keras. Their combination has tendency to create new models. New classes and functions can easily consume  these models . Rather than as separate model configuration files  Python code contains these Models.

The main reasons for the use of Keras lie in the design principles followed, above all the focus on user friendliness. It is easy to learn and models are easy to build. In addition, Keras offers the advantages of a large user base, supports a wide range of deployment options, multiple GPUs and distributed training. Google, Microsoft, Amazon, Apple, Nvidia, Uber and many others support the tool.

Pruning API

The Google Pruning API optimization tool is technically based on Keras. Therefore, it should be easy to integrate into existing Keras projects. The intention behind is to optimize machine learning models already in the training phase.

According to the name, the API trims ML models. During training, she evaluates the connections between the different levels of the model. There are no Unimportant or irrelevant connections in the network. This reduces both the memory capacity required to save the model and the RAM required to run the model, as well as the CPU operations required.

Horovod

Horovod is a distributed training framework. Uber develop it for TensorFlow, Keras and the open source program library PyTorch. Horovod should make distributed deep learning quick and easy to use. It is based on ideas from Baidu’s experimental implementation of the TensorFlow Ring Allreduce algorithm.

Uber originally tried to use Distributed TensorFlow with parameter servers. The engineers found that the message passing interface (MPI) model was less complicated and needed less code adjustments. Uber claims that the Horovod system enables an AI model which can be trained 2 times faster traditional TensorFlow implementation.

Horovod uses Open MPI (or another MPI implementation) to exchange messages between nodes, and Nvidia NCCL for its highly optimized version of Ring Allreduce. Horovod achieves 90 percent scaling efficiency for Inception-v3 and ResNet-101 . For VGG-16 , it achieves a scaling efficiency of 68 percent on up to 512 Nvidia Pascal GPUs.

In December 2018, Uber announced that it was sponsoring the Horovod project under the aegis of the LF Deep Learning Foundation for open source AI software from the Linux Foundation.

The Tony Project

LinkedIn made the code for their Tony project public in late 2018 . According to Serdar Yegulalp from InfoWorld , the open source tool is used to manage and scale deep learning jobs in TensorFlow. It uses the YARN (Yet Another Resource Negotiator) job planning system in Hadoop.

While there are already some other planning tools, LinkedIn found a few limitations. TensorFlow on Spark, for example, operates the framework on the Apache Spark job engine. Actually it is therefore very similar to Spark. TensorFlowOnYARN offers the same basic features as Tony. Fault tolerance is not its part.

According to LinkedIn, Tony uses YARN’s resource and task planning system to set up TensorFlow jobs in a Hadoop cluster. The tool should also make it possible

• Schedule GPU-based TensorFlow jobs through Hadoop;
• request different types of resources (CPUs or GPUs);
• Allocate memory differently for TensorFlow nodes;
• Store job results regularly in the Hadoop Distributed File System (HDFS) and resume at one point if they are interrupted or crash.

Tony divides the work into three different internal components: one Client, an application master and one who performs the task. The client receives incoming TensorFlow jobs. The master coordinates with YARN’s resource manager how the job should be provisioned in YARN. The executing component is what is actually running on the YARN cluster to process the TensorFlow job.

According to LinkedIn, Tony has no noticeable overhead for TensorFlow because it is in the layer that orchestrates distributed TensorFlow. Therefore, it does not affect the actual execution of the TensorFlow job.

Tony also works with the TensorBoard app to visualize TensorFlow, optimize it and free it from errors (debugging).

Inkling

Inkling is a commercial high-level programming language from Bonsai (now a Microsoft subsidiary ) that makes it easier to build AI applications. According to Paul Krill from InfoWorld , it compiles down to the TensorFlow library. Inkling is designed to present AI in a way that programmers can focus on teaching something to a system rather than focusing on the low-level mechanics.

Inkling abstracts dynamic AI algorithms that normally require specialist knowledge in the field of machine learning . According to Bonsai, the language is descriptive and the syntax is reminiscent of a mixture of Python and SQL. The aim of the language is to make machine learning accessible to developers and engineers who have no ML background but want to use the technology in their respective specialist areas.

Tensor2Tensor

Google’s open source project Tensor2Tensor (T2T) should reduce the workload to configure a deep learning model for the training. It is a Python based library for Workflow-Optimization of TensorFlow training tasks. This allows developers to specify the key elements in a TensorFlow model and define their relationships with one another.

The key elements according to InfoWorld editor Serdar Yegulalp are:

• Records : T2T support from already numerous records for training. New data records can be customized Workflow or add to the core T2T project using a pull request .
• Problems and modalities : These describe which tasks are being trained for (such as speech recognition or translation) and which data are to be received and generated from them. For example, an image recognition system would receive image data and output text descriptions.
• Models : Many common models are already registered with T2T and more can be added.
• Hyper parameter : A hyper parameter is a parameter that is used to control the ML training algorithm and, unlike other parameters, its value is not learned in the actual training of the model. Different sets of settings that control the training process can be created in T2T. These can be changed or interconnected as required.
• Trainer : The parameters that are passed to the actual training binary can be specified separately.

T2T provides presets for each element. Numerous common models and data sets are already included. This allows training to begin quickly by reusing or expanding it. What T2T doesn’t deliver is a broader context beyond TensorFlow, how the deep learning project should be organized. It “only” makes it easier to use the framework.

Ludwig

Another tool that is supposed to simplify deep learning training comes as open source from the Horovod makers Uber. When we use  Ludwig there is no programming effort. Training and testing is possible ML models

According to Experts, the tool takes a data type-related approach to model development, with specific encoders and decoders for each data type. However, different encoders and decoders should be used for each type. In the model configuration file (in YAML format) we can set the desired encoder and hyper parameters without having to write code. In the current version, Ludwig offers encoders and decoders for common data types such as binary values, floating point numbers, categories, discrete sequences, images, texts and time series, which should be supplemented by pre-trained models if necessary.

TensorWatch

With TensorWatch published Microsoft Research an open source – debugging tool for machine learning that will help even the most complex problems.

Log files are not created during model training. The reason is that it depends on the size of the data records, they can cause high storage costs. However, this means that there is no overall overview of errors in the model.

TensorWatch is to visualize interactive real-time debugging in Jupyter Notebook as well as offer custom UIs and the possibility of integration into the Python ecosystem. Jupyter Notebook is an open source web application. share documents that contain live code, formulas, visualizations its product. Microsoft provides bar graphs, histograms, pie charts and 3D variations as supported visualization types.

in TensorWatch treats Data and other objects as streams – including files, consoles, sockets, Cloud-Storage and the visualizations themselves. These streams can “listen” to other streams so that, for example, several of them can be combined in one visualization or one stream in several visualizations at the same time.

With the “Lazy Logging Mode” TensorWatch can monitor the variables of the model without explicit logging. Users can make interactive requests in real time that run in the context of these variables and return a stream as a result. visualizing, storage and processing is possible for stream.

MorphNet

Google’s open source MorphNet TensorFlow extension is a tool to improve deep neural networks by adapting the network architecture to new requirements. To do this, the tool analyzes the neural networks and then optimizes them by changing their structure.

According to JAXenter , the tool works in two steps:

• Shrinkage : MorphNet analyzes the structure of a network. It also measure the efficiency of individual neurons According to some criteria. Neurons rated as inefficient are not the part of network.
• The extension : After removing the inefficient neurons, the tool expands all layers of the network by an identical factor. Efficient layers, from which fewer neurons have been removed, increase in size relatively. The expansion is less than the shrinkage, saving resources while increasing efficiency and maintaining accuracy.

Shortlink:

Posted by Paracha on July 21, 2020. Filed under Articles, Artificial Intelligence, Technology. You can follow any responses to this entry through the RSS 2.0. You can skip to the end and leave a response. Pinging is currently not allowed.