Christopher Maier
Google Vertex AI enables the efficient development, deployment, and management of Machine Learning models. The wiki provides a guide for a comprehensive overview of the platform's key functions, including Model Registry, Online and Batch Predictions, and Vector Search.
The Google Vertex AI tool is crucial for optimizing ML workflows, accelerating data processing, and improving result accuracy. The platform is particularly suitable for scaling and managing AI projects across various business areas.
Table of contents
- In-depth insights into Google Vertex AI
- 1.1 Dashboard
- In-depth insights into Google Vertex AI
- 1.2 Model Garden
- 2. Overview of the most important models and APIs in Google Vertex AI: Possible uses and applications
- 3. Notebooks for Data Science and Machine Learning
- 4. AI Studio
- 5. Efficient data management for ML models
In-depth insights into Google Vertex AI
1. Activate APIs and pre-built models: Optimal use of Google Vertex AI
1.1 Dashboard
Vertex AI is a comprehensive platform that provides a variety of APIs to enable the use of artificial intelligence in various applications. To fully utilize Vertex AI, the following APIs must be enabled:
In addition, Vertex AI enables the monitoring of specific metrics of individual services to effectively monitor performance and usage. This provides valuable insights into the operational aspects of the platform and helps to optimize applications.
In-depth insights into Google Vertex AI
1.2 Model Garden
Model Garden is a collection of pre-built machine learning models and tools available in Google Cloud Vertex AI. It enables developers and data scientists to quickly and easily create, test, and deploy models for a variety of use cases. There are a total of 153 models available.
The models are divided into three main categories:
- Broad knowledge: Versatile models trained on large amounts of data.
- Adaptable: Can be optimized and specialized for various tasks.
- Specialization: Refined versions of Foundation Models, adapted to specific tasks.
- Extended customization: Requires additional data and code for precise customization and performance enhancement.
- Ready to use: Pre-built models for specific tasks, ready for immediate use.
- Possible fine-tuning: Some solutions allow adjustments with your own data for individual optimization.
In addition, the models can be further divided into categories:
- Language Models
- Tabular Models
- Document Models
- Multimodal Models
- Vision Models
- Speech Models
- Video Model
Another way to categorize models is by their features:
- Vertex AI Studio (ready-to-use models that can be tested in the console, such as Gemini)
- API available (models that can be called by an API)
- Open source
- Notebook support
- Pipeline support
- One-click-deployement
Gemini 1.0 Pro: Gemini 1.0 Pro is the name of the large language model Gemini that understands language and it is a base model suitable for a variety of natural language processing tasks, such as summarization, instruction following, content generation, sentiment analysis, entity extraction, classification, etc. Content that Gemini 1.0 Pro can generate includes document summaries, answers to questions, labels for classifying content, and much more.
Use Cases:
- Question answering
- Classification
- Sentiment analysis
- Entity extraction
Average Word Embedding Classifier: The Average Word Embedding Classifier is a simple, lightweight text classification model that uses average word vectors to create a text representation and evaluates it via a two-layer network with RELU activation. The model is designed for use on devices and can be used with MediaPipe Tasks TextClassifier on various platforms such as Android, iOS, Web and Desktop.
Use Cases:
- Text classification
Text Moderation: Text moderation analyzes a document and identifies harmful and sensitive categories that apply to the text found in the document.
Use Cases:
- Checking for potentially offensive content that could lead to a negative perception of the brand.
- Ensure that user-generated and editorial content is brand-safe for advertising.
- Identification of toxic comments in forums and chat messages that could offend and deter users.
2.2 Tabular:
Claude 3 Opus: Claude 3 Opus from Anthropic is the company's most powerful AI model and demonstrates outstanding capabilities in complex tasks. The model can handle open tasks and unknown scenarios with impressive eloquence and human-like understanding. All Claude 3 models can process images, output text and have a 200K K
Use Cases:
- Task automation: Planning and execution of complex actions via APIs and databases, interactive P
- Research and Development (R&D): Review of research papers, brainstorming and hypothesis generation, drug research.
- Strategy: Advanced analysis of charts and graphs, financial and market trends, forecasts.
- Visual capabilities: Image and text processing for analyzing and understanding diagrams, graphics, technical drawings, reports, and other visual media.
AutoGluon: AutoGluon automates machine learning processes to achieve high prediction performance in your applications. However, this version only supports tabular data, allowing you to train and implement highly accurate machine learning and deep learning models for such data. With a simple call to fit(), you can achieve high accuracy in standard supervised learning tasks (classification and regression) without the complexities of data cleaning, feature extraction, hyperparameter optimization, and model selection.
Use Cases:
- Prediction of tabular data: Predicting the values of a target column based on the other columns of a tabular dataset.
AutoML E2E: The "Tabular Workflow for End-to-End AutoML" is a complete AutoML pipeline for classification and regression tasks, similar to the AutoML API, but offers more control over the individual process steps. The pipeline includes automated feature engineering, architecture search, and hyperparameter tuning. Successful models go through several stages, including cross-validation and model combination, to optimize the final model selection. The AutoML solution was successfully tested in a competition and achieved second place.
Use Cases:
- Marketing and customer analysis: Estimation of order frequency, probability of customer churn, conversion probability of leads, customer lifetime value, campaign assignment.
- Resource Utilization: Probability of equipment failures, estimation of supply and demand for drivers, employee turnover, life expectancy of equipment.
- Risk Management: Estimation of damage amount and probability, probability of fraud, probability of failure.
- Ranking: Optimal product placement and advertising effectiveness.
2.3 Document:
Claude 3 Opus: see the following section
Document AI OCR Processor: Document OCR recognizes and extracts text from documents in over 200 printed and 50 handwritten languages. It identifies text blocks, paragraphs, lines, words, and optionally symbols in PDFs and images and can automatically straighten documents to increase accuracy. Additionally, it enables the recognition of font styles, language hints, and the evaluation of image quality for optimized further processing.
Use Cases:
- Document Digitalization: Digitize text from documents, automate data entry, improve and verify data quality, programmatic preprocessing.
- Invoice and Claims Processing: Extraction of data from invoices to automate invoice and claims processes and reduce errors.
- Document Search & Q&A: Extract data from documents to enable automated processes for questions and search functions.
- Contract management: Extract data from contracts to increase their accessibility and searchability and to support the correct execution of contracts.
- Archiving: Convert paper documents into electronic formats to improve document accessibility.
- Compliance: Extract data from documents to ensure compliance and perform content-based moderation.
2.4 Multimodal:
Gemini 1.5 Pro: The Gemini 1.5 Pro is a versatile base model that is particularly suitable for multimodal tasks such as visual understanding, classifying, summarizing, and generating content from images, audios, and videos. It efficiently processes visual and textual inputs such as photos, documents, infographics, and screenshots. The model is part of the Gemini model family, which offers various sizes and capabilities, including specialized versions for text and visual content.
Use Cases:
- Visual information retrieval: Combining external knowledge with information extracted from images or videos to answer questions.
- Object Recognition: Detailed identification of objects in images and videos.
- Digital Content Understanding: Extraction of information from visual content such as infographics and websites.
- Structured content production: Generating answers based on multimodal inputs in formats such as HTML and JSON.
- Description and subtitling: Creation of descriptions for images and videos in various
- Inferences: Novel information creation without memorization or retrieval.
- Audio: Analysis of voice files for summaries, transcriptions and Q&A.
- Multimodal processing: Simultaneous processing of different media types such as video and audio.
2.5 Vision, Speech and Video Models
These are included in the white paper.
3. Notebooks for Data Science and Machine Learning
Notebooks are interactive environments that allow the integration of code, text, and visualizations in one document. These environments are frequently used in the fields of data science and machine learning, especially for exploratory data analysis, model prototyping, and communication of results.
3.1 Colab Enterprise
Colab Enterprise is a notebook environment based on the Google Cloud platform, specifically designed for companies to provide a scalable and secure platform for collaborating on data science projects. This environment supports the use of powerful GPUs and CPUs to meet the requirements of various projects. Colab Enterprise also offers security mechanisms for data processing and storage and enables real-time collaboration between multiple users on a project. In addition, seamless integration with other Google Cloud Services, such as BigQuery and Cloud Storage, is possible.
3.2 Workbench
Workbench is an open-source notebook platform based on Jupyter that can be operated both locally on a computer and in the cloud. The platform is highly customizable and expandable through a variety of plugins and extensions. It offers a flexible working environment that can be tailored to the specific needs and preferences of data scientists. Since Workbench is an open-source project, the source code is freely available and can be used and modified by users free of charge.
4. AI Studio
4.1 Overview
Vertex AI Studio is a web-based development environment for Machine Learning (ML) on the Google Cloud Platform (GCP). It enables teams of data scientists, ML engineers, and business users to create, train, and deploy ML models without having to write code.
Vertex AI Studio offers a range of features that simplify the development of ML models, including:
- A drag-and-drop interface for creating ML pipelines
- Pre-trained models for various tasks such as image classification and natural language processing
- Tools for training and optimizing models
- Ways to deploy models in production
Vertex AI Studio is a great choice for teams looking to quickly and easily develop and deploy machine learning models. It's also a good option for teams new to machine learning, as it offers a user-friendly interface and a range of pre-built features.
4.2 Multimodal
Vertex AI Studio supports the development of multimodal ML models that can process data from various sources like images, text, and audio. This allows for the development of more powerful and versatile ML models.
Vertex AI Studio offers a range of features that support the development of multimodal models:
- Data Preprocessing Tools for merging and preparing data from different modalities
- Models for multimodal tasks such as image-text assignment and audio-video recognition
- Tools for evaluating the performance of multimodal models
Vertex AI Studio is a good choice for teams wanting to develop multimodal machine learning models. It offers several features that simplify the development and deployment of these models.
4.3 Language
Vertex AI Studio supports the development of language models that can process and generate text. This enables the development of applications like chatbots, machine translation, and text generation. Vertex AI Studio offers several features to support the development of language models:
- Pre-trained language models for various languages
- Tools for training speech models on custom data
- Tools for evaluating the performance of speech models
Vertex AI Studio is a good choice for teams wanting to develop language models. It offers several features that simplify the development and deployment of these models.
4.4 Vision
Vertex AI Studio supports the development of vision models that can process images and videos. This enables the development of applications like image classification, object recognition, and image segmentation.
Vertex AI Studio provides several features to support the development of image processing models:
- Pre-trained Vision models for various tasks such as image classification and object recognition
- Tools for training vision models on custom data
- Tools for evaluating the performance of vision models
Vertex AI Studio is a good choice for teams wanting to develop vision models. It offers several features that simplify the development and deployment of these models.
4.5 Speech
Vertex AI Studio supports the development of speech models that can process and generate speech. This enables the development of applications like speech recognition, speech synthesis, and language translation. Vertex AI Studio offers several features to support the development of speech models, including:
- Pre-trained Speech models for various languages
- Tools for training speech models on custom data
- Tools for evaluating the performance of speech models
Vertex AI Studio is a good choice for teams looking to develop speech models. It provides several features that make developing and deploying these models easier. Note: Speech is not yet fully supported in the current version of Vertex AI Studio. However, it is under development and will be available soon.
5. Efficient data management for ML models
5.1 Feature Store
The Feature Store is a central repository for managing features for Machine Learning (ML) models. It enables data scientists and ML engineers to store, version, and reuse features in one place, simplifying the development and deployment of ML models. The Feature Store offers a number of functions that simplify the management of features, including: Storage of features in various formats, including numerical data, categorical data, and text data Versioning of features to track changes to features over time Reuse of features in multiple ML models Search and filter functions to find the required features Integration with Vertex AI to simplify the development and deployment of ML models.
A Feature Store is a great choice for teams using multiple ML models who want to manage their features centrally. It can help speed up the development and deployment of ML models and improve their quality.
- Storage of characteristics in various formats, including numerical data, categorical data and text data
- Versioning of characteristics to track changes to characteristics over time
- Reuse of features in multiple ML models
- Search and filter functions to find the required features
- Integration with Vertex AI to simplify the development and deployment of ML models
A Feature Store is a great choice for teams using multiple ML models who want to manage their features centrally. It can help speed up the development and deployment of ML models and improve their quality.
In-depth insights into Google Vertex AI
Discover the full potential of Google Vertex AI for your company. Our experts are ready to support you in implementing this powerful platform and developing customized AI solutions for your specific challenges.
Schedule a free consultation now and learn how you can use Google Vertex AI to:
- Optimize your ML workflows
- Accelerate the development of AI models
- Make data-driven decisions
- Give your company a competitive edge
Let's shape the future of your company with AI together. Contact us today and take the first step towards AI-powered innovation!
