feat: Add Official Microsoft & Gemini Skills (845+ Total)
🚀 Impact Significantly expands the capabilities of **Antigravity Awesome Skills** by integrating official skill collections from **Microsoft** and **Google Gemini**. This update increases the total skill count to **845+**, making the library even more comprehensive for AI coding assistants. ✨ Key Changes 1. New Official Skills - **Microsoft Skills**: Added a massive collection of official skills from [microsoft/skills](https://github.com/microsoft/skills). - Includes Azure, .NET, Python, TypeScript, and Semantic Kernel skills. - Preserves the original directory structure under `skills/official/microsoft/`. - Includes plugin skills from the `.github/plugins` directory. - **Gemini Skills**: Added official Gemini API development skills under `skills/gemini-api-dev/`. 2. New Scripts & Tooling - **`scripts/sync_microsoft_skills.py`**: A robust synchronization script that: - Clones the official Microsoft repository. - Preserves the original directory heirarchy. - Handles symlinks and plugin locations. - Generates attribution metadata. - **`scripts/tests/inspect_microsoft_repo.py`**: Debug tool to inspect the remote repository structure. - **`scripts/tests/test_comprehensive_coverage.py`**: Verification script to ensure 100% of skills are captured during sync. 3. Core Improvements - **`scripts/generate_index.py`**: Enhanced frontmatter parsing to safely handle unquoted values containing `@` symbols and commas (fixing issues with some Microsoft skill descriptions). - **`package.json`**: Added `sync:microsoft` and `sync:all-official` scripts for easy maintenance. 4. Documentation - Updated `README.md` to reflect the new skill counts (845+) and added Microsoft/Gemini to the provider list. - Updated `CATALOG.md` and `skills_index.json` with the new skills. 🧪 Verification - Ran `scripts/tests/test_comprehensive_coverage.py` to verify all Microsoft skills are detected. - Validated `generate_index.py` fixes by successfully indexing the new skills.
This commit is contained in:
Ahmed Rehan
256
skills/official/microsoft/java/foundry/anomalydetector/SKILL.md
Normal file
256
skills/official/microsoft/java/foundry/anomalydetector/SKILL.md
Normal file
@@ -0,0 +1,256 @@
|
||||
---
|
||||
name: azure-ai-anomalydetector-java
|
||||
description: Build anomaly detection applications with Azure AI Anomaly Detector SDK for Java. Use when implementing univariate/multivariate anomaly detection, time-series analysis, or AI-powered monitoring.
|
||||
package: com.azure:azure-ai-anomalydetector
|
||||
---
|
||||
|
||||
# Azure AI Anomaly Detector SDK for Java
|
||||
|
||||
Build anomaly detection applications using the Azure AI Anomaly Detector SDK for Java.
|
||||
|
||||
## Installation
|
||||
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>com.azure</groupId>
|
||||
<artifactId>azure-ai-anomalydetector</artifactId>
|
||||
<version>3.0.0-beta.6</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
## Client Creation
|
||||
|
||||
### Sync and Async Clients
|
||||
|
||||
```java
|
||||
import com.azure.ai.anomalydetector.AnomalyDetectorClientBuilder;
|
||||
import com.azure.ai.anomalydetector.MultivariateClient;
|
||||
import com.azure.ai.anomalydetector.UnivariateClient;
|
||||
import com.azure.core.credential.AzureKeyCredential;
|
||||
|
||||
String endpoint = System.getenv("AZURE_ANOMALY_DETECTOR_ENDPOINT");
|
||||
String key = System.getenv("AZURE_ANOMALY_DETECTOR_API_KEY");
|
||||
|
||||
// Multivariate client for multiple correlated signals
|
||||
MultivariateClient multivariateClient = new AnomalyDetectorClientBuilder()
|
||||
.credential(new AzureKeyCredential(key))
|
||||
.endpoint(endpoint)
|
||||
.buildMultivariateClient();
|
||||
|
||||
// Univariate client for single variable analysis
|
||||
UnivariateClient univariateClient = new AnomalyDetectorClientBuilder()
|
||||
.credential(new AzureKeyCredential(key))
|
||||
.endpoint(endpoint)
|
||||
.buildUnivariateClient();
|
||||
```
|
||||
|
||||
### With DefaultAzureCredential
|
||||
|
||||
```java
|
||||
import com.azure.identity.DefaultAzureCredentialBuilder;
|
||||
|
||||
MultivariateClient client = new AnomalyDetectorClientBuilder()
|
||||
.credential(new DefaultAzureCredentialBuilder().build())
|
||||
.endpoint(endpoint)
|
||||
.buildMultivariateClient();
|
||||
```
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### Univariate Anomaly Detection
|
||||
- **Batch Detection**: Analyze entire time series at once
|
||||
- **Streaming Detection**: Real-time detection on latest data point
|
||||
- **Change Point Detection**: Detect trend changes in time series
|
||||
|
||||
### Multivariate Anomaly Detection
|
||||
- Detect anomalies across 300+ correlated signals
|
||||
- Uses Graph Attention Network for inter-correlations
|
||||
- Three-step process: Train → Inference → Results
|
||||
|
||||
## Core Patterns
|
||||
|
||||
### Univariate Batch Detection
|
||||
|
||||
```java
|
||||
import com.azure.ai.anomalydetector.models.*;
|
||||
import java.time.OffsetDateTime;
|
||||
import java.util.List;
|
||||
|
||||
List<TimeSeriesPoint> series = List.of(
|
||||
new TimeSeriesPoint(OffsetDateTime.parse("2023-01-01T00:00:00Z"), 1.0),
|
||||
new TimeSeriesPoint(OffsetDateTime.parse("2023-01-02T00:00:00Z"), 2.5),
|
||||
// ... more data points (minimum 12 points required)
|
||||
);
|
||||
|
||||
UnivariateDetectionOptions options = new UnivariateDetectionOptions(series)
|
||||
.setGranularity(TimeGranularity.DAILY)
|
||||
.setSensitivity(95);
|
||||
|
||||
UnivariateEntireDetectionResult result = univariateClient.detectUnivariateEntireSeries(options);
|
||||
|
||||
// Check for anomalies
|
||||
for (int i = 0; i < result.getIsAnomaly().size(); i++) {
|
||||
if (result.getIsAnomaly().get(i)) {
|
||||
System.out.printf("Anomaly detected at index %d with value %.2f%n",
|
||||
i, series.get(i).getValue());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Univariate Last Point Detection (Streaming)
|
||||
|
||||
```java
|
||||
UnivariateLastDetectionResult lastResult = univariateClient.detectUnivariateLastPoint(options);
|
||||
|
||||
if (lastResult.isAnomaly()) {
|
||||
System.out.println("Latest point is an anomaly!");
|
||||
System.out.printf("Expected: %.2f, Upper: %.2f, Lower: %.2f%n",
|
||||
lastResult.getExpectedValue(),
|
||||
lastResult.getUpperMargin(),
|
||||
lastResult.getLowerMargin());
|
||||
}
|
||||
```
|
||||
|
||||
### Change Point Detection
|
||||
|
||||
```java
|
||||
UnivariateChangePointDetectionOptions changeOptions =
|
||||
new UnivariateChangePointDetectionOptions(series, TimeGranularity.DAILY);
|
||||
|
||||
UnivariateChangePointDetectionResult changeResult =
|
||||
univariateClient.detectUnivariateChangePoint(changeOptions);
|
||||
|
||||
for (int i = 0; i < changeResult.getIsChangePoint().size(); i++) {
|
||||
if (changeResult.getIsChangePoint().get(i)) {
|
||||
System.out.printf("Change point at index %d with confidence %.2f%n",
|
||||
i, changeResult.getConfidenceScores().get(i));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Multivariate Model Training
|
||||
|
||||
```java
|
||||
import com.azure.ai.anomalydetector.models.*;
|
||||
import com.azure.core.util.polling.SyncPoller;
|
||||
|
||||
// Prepare training request with blob storage data
|
||||
ModelInfo modelInfo = new ModelInfo()
|
||||
.setDataSource("https://storage.blob.core.windows.net/container/data.zip?sasToken")
|
||||
.setStartTime(OffsetDateTime.parse("2023-01-01T00:00:00Z"))
|
||||
.setEndTime(OffsetDateTime.parse("2023-06-01T00:00:00Z"))
|
||||
.setSlidingWindow(200)
|
||||
.setDisplayName("MyMultivariateModel");
|
||||
|
||||
// Train model (long-running operation)
|
||||
AnomalyDetectionModel trainedModel = multivariateClient.trainMultivariateModel(modelInfo);
|
||||
|
||||
String modelId = trainedModel.getModelId();
|
||||
System.out.println("Model ID: " + modelId);
|
||||
|
||||
// Check training status
|
||||
AnomalyDetectionModel model = multivariateClient.getMultivariateModel(modelId);
|
||||
System.out.println("Status: " + model.getModelInfo().getStatus());
|
||||
```
|
||||
|
||||
### Multivariate Batch Inference
|
||||
|
||||
```java
|
||||
MultivariateBatchDetectionOptions detectionOptions = new MultivariateBatchDetectionOptions()
|
||||
.setDataSource("https://storage.blob.core.windows.net/container/inference-data.zip?sasToken")
|
||||
.setStartTime(OffsetDateTime.parse("2023-07-01T00:00:00Z"))
|
||||
.setEndTime(OffsetDateTime.parse("2023-07-31T00:00:00Z"))
|
||||
.setTopContributorCount(10);
|
||||
|
||||
MultivariateDetectionResult detectionResult =
|
||||
multivariateClient.detectMultivariateBatchAnomaly(modelId, detectionOptions);
|
||||
|
||||
String resultId = detectionResult.getResultId();
|
||||
|
||||
// Poll for results
|
||||
MultivariateDetectionResult result = multivariateClient.getBatchDetectionResult(resultId);
|
||||
for (AnomalyState state : result.getResults()) {
|
||||
if (state.getValue().isAnomaly()) {
|
||||
System.out.printf("Anomaly at %s, severity: %.2f%n",
|
||||
state.getTimestamp(),
|
||||
state.getValue().getSeverity());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Multivariate Last Point Detection
|
||||
|
||||
```java
|
||||
MultivariateLastDetectionOptions lastOptions = new MultivariateLastDetectionOptions()
|
||||
.setVariables(List.of(
|
||||
new VariableValues("variable1", List.of("timestamp1"), List.of(1.0f)),
|
||||
new VariableValues("variable2", List.of("timestamp1"), List.of(2.5f))
|
||||
))
|
||||
.setTopContributorCount(5);
|
||||
|
||||
MultivariateLastDetectionResult lastResult =
|
||||
multivariateClient.detectMultivariateLastAnomaly(modelId, lastOptions);
|
||||
|
||||
if (lastResult.getValue().isAnomaly()) {
|
||||
System.out.println("Anomaly detected!");
|
||||
// Check contributing variables
|
||||
for (AnomalyContributor contributor : lastResult.getValue().getInterpretation()) {
|
||||
System.out.printf("Variable: %s, Contribution: %.2f%n",
|
||||
contributor.getVariable(),
|
||||
contributor.getContributionScore());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Model Management
|
||||
|
||||
```java
|
||||
// List all models
|
||||
PagedIterable<AnomalyDetectionModel> models = multivariateClient.listMultivariateModels();
|
||||
for (AnomalyDetectionModel m : models) {
|
||||
System.out.printf("Model: %s, Status: %s%n",
|
||||
m.getModelId(),
|
||||
m.getModelInfo().getStatus());
|
||||
}
|
||||
|
||||
// Delete a model
|
||||
multivariateClient.deleteMultivariateModel(modelId);
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
```java
|
||||
import com.azure.core.exception.HttpResponseException;
|
||||
|
||||
try {
|
||||
univariateClient.detectUnivariateEntireSeries(options);
|
||||
} catch (HttpResponseException e) {
|
||||
System.out.println("Status code: " + e.getResponse().getStatusCode());
|
||||
System.out.println("Error: " + e.getMessage());
|
||||
}
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
AZURE_ANOMALY_DETECTOR_ENDPOINT=https://<resource>.cognitiveservices.azure.com/
|
||||
AZURE_ANOMALY_DETECTOR_API_KEY=<your-api-key>
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Minimum Data Points**: Univariate requires at least 12 points; more data improves accuracy
|
||||
2. **Granularity Alignment**: Match `TimeGranularity` to your actual data frequency
|
||||
3. **Sensitivity Tuning**: Higher values (0-99) detect more anomalies
|
||||
4. **Multivariate Training**: Use 200-1000 sliding window based on pattern complexity
|
||||
5. **Error Handling**: Always handle `HttpResponseException` for API errors
|
||||
|
||||
## Trigger Phrases
|
||||
|
||||
- "anomaly detection Java"
|
||||
- "detect anomalies time series"
|
||||
- "multivariate anomaly Java"
|
||||
- "univariate anomaly detection"
|
||||
- "streaming anomaly detection"
|
||||
- "change point detection"
|
||||
- "Azure AI Anomaly Detector"
|
||||
282
skills/official/microsoft/java/foundry/contentsafety/SKILL.md
Normal file
282
skills/official/microsoft/java/foundry/contentsafety/SKILL.md
Normal file
@@ -0,0 +1,282 @@
|
||||
---
|
||||
name: azure-ai-contentsafety-java
|
||||
description: Build content moderation applications with Azure AI Content Safety SDK for Java. Use when implementing text/image analysis, blocklist management, or harm detection for hate, violence, sexual content, and self-harm.
|
||||
package: com.azure:azure-ai-contentsafety
|
||||
---
|
||||
|
||||
# Azure AI Content Safety SDK for Java
|
||||
|
||||
Build content moderation applications using the Azure AI Content Safety SDK for Java.
|
||||
|
||||
## Installation
|
||||
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>com.azure</groupId>
|
||||
<artifactId>azure-ai-contentsafety</artifactId>
|
||||
<version>1.1.0-beta.1</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
## Client Creation
|
||||
|
||||
### With API Key
|
||||
|
||||
```java
|
||||
import com.azure.ai.contentsafety.ContentSafetyClient;
|
||||
import com.azure.ai.contentsafety.ContentSafetyClientBuilder;
|
||||
import com.azure.ai.contentsafety.BlocklistClient;
|
||||
import com.azure.ai.contentsafety.BlocklistClientBuilder;
|
||||
import com.azure.core.credential.KeyCredential;
|
||||
|
||||
String endpoint = System.getenv("CONTENT_SAFETY_ENDPOINT");
|
||||
String key = System.getenv("CONTENT_SAFETY_KEY");
|
||||
|
||||
ContentSafetyClient contentSafetyClient = new ContentSafetyClientBuilder()
|
||||
.credential(new KeyCredential(key))
|
||||
.endpoint(endpoint)
|
||||
.buildClient();
|
||||
|
||||
BlocklistClient blocklistClient = new BlocklistClientBuilder()
|
||||
.credential(new KeyCredential(key))
|
||||
.endpoint(endpoint)
|
||||
.buildClient();
|
||||
```
|
||||
|
||||
### With DefaultAzureCredential
|
||||
|
||||
```java
|
||||
import com.azure.identity.DefaultAzureCredentialBuilder;
|
||||
|
||||
ContentSafetyClient client = new ContentSafetyClientBuilder()
|
||||
.credential(new DefaultAzureCredentialBuilder().build())
|
||||
.endpoint(endpoint)
|
||||
.buildClient();
|
||||
```
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### Harm Categories
|
||||
| Category | Description |
|
||||
|----------|-------------|
|
||||
| Hate | Discriminatory language based on identity groups |
|
||||
| Sexual | Sexual content, relationships, acts |
|
||||
| Violence | Physical harm, weapons, injury |
|
||||
| Self-harm | Self-injury, suicide-related content |
|
||||
|
||||
### Severity Levels
|
||||
- Text: 0-7 scale (default outputs 0, 2, 4, 6)
|
||||
- Image: 0, 2, 4, 6 (trimmed scale)
|
||||
|
||||
## Core Patterns
|
||||
|
||||
### Analyze Text
|
||||
|
||||
```java
|
||||
import com.azure.ai.contentsafety.models.*;
|
||||
|
||||
AnalyzeTextResult result = contentSafetyClient.analyzeText(
|
||||
new AnalyzeTextOptions("This is text to analyze"));
|
||||
|
||||
for (TextCategoriesAnalysis category : result.getCategoriesAnalysis()) {
|
||||
System.out.printf("Category: %s, Severity: %d%n",
|
||||
category.getCategory(),
|
||||
category.getSeverity());
|
||||
}
|
||||
```
|
||||
|
||||
### Analyze Text with Options
|
||||
|
||||
```java
|
||||
AnalyzeTextOptions options = new AnalyzeTextOptions("Text to analyze")
|
||||
.setCategories(Arrays.asList(
|
||||
TextCategory.HATE,
|
||||
TextCategory.VIOLENCE))
|
||||
.setOutputType(AnalyzeTextOutputType.EIGHT_SEVERITY_LEVELS);
|
||||
|
||||
AnalyzeTextResult result = contentSafetyClient.analyzeText(options);
|
||||
```
|
||||
|
||||
### Analyze Text with Blocklist
|
||||
|
||||
```java
|
||||
AnalyzeTextOptions options = new AnalyzeTextOptions("I h*te you and want to k*ll you")
|
||||
.setBlocklistNames(Arrays.asList("my-blocklist"))
|
||||
.setHaltOnBlocklistHit(true);
|
||||
|
||||
AnalyzeTextResult result = contentSafetyClient.analyzeText(options);
|
||||
|
||||
if (result.getBlocklistsMatch() != null) {
|
||||
for (TextBlocklistMatch match : result.getBlocklistsMatch()) {
|
||||
System.out.printf("Blocklist: %s, Item: %s, Text: %s%n",
|
||||
match.getBlocklistName(),
|
||||
match.getBlocklistItemId(),
|
||||
match.getBlocklistItemText());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Analyze Image
|
||||
|
||||
```java
|
||||
import com.azure.ai.contentsafety.models.*;
|
||||
import com.azure.core.util.BinaryData;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Paths;
|
||||
|
||||
// From file
|
||||
byte[] imageBytes = Files.readAllBytes(Paths.get("image.png"));
|
||||
ContentSafetyImageData imageData = new ContentSafetyImageData()
|
||||
.setContent(BinaryData.fromBytes(imageBytes));
|
||||
|
||||
AnalyzeImageResult result = contentSafetyClient.analyzeImage(
|
||||
new AnalyzeImageOptions(imageData));
|
||||
|
||||
for (ImageCategoriesAnalysis category : result.getCategoriesAnalysis()) {
|
||||
System.out.printf("Category: %s, Severity: %d%n",
|
||||
category.getCategory(),
|
||||
category.getSeverity());
|
||||
}
|
||||
```
|
||||
|
||||
### Analyze Image from URL
|
||||
|
||||
```java
|
||||
ContentSafetyImageData imageData = new ContentSafetyImageData()
|
||||
.setBlobUrl("https://example.com/image.jpg");
|
||||
|
||||
AnalyzeImageResult result = contentSafetyClient.analyzeImage(
|
||||
new AnalyzeImageOptions(imageData));
|
||||
```
|
||||
|
||||
## Blocklist Management
|
||||
|
||||
### Create or Update Blocklist
|
||||
|
||||
```java
|
||||
import com.azure.core.http.rest.RequestOptions;
|
||||
import com.azure.core.http.rest.Response;
|
||||
import com.azure.core.util.BinaryData;
|
||||
import java.util.Map;
|
||||
|
||||
Map<String, String> description = Map.of("description", "Custom blocklist");
|
||||
BinaryData resource = BinaryData.fromObject(description);
|
||||
|
||||
Response<BinaryData> response = blocklistClient.createOrUpdateTextBlocklistWithResponse(
|
||||
"my-blocklist", resource, new RequestOptions());
|
||||
|
||||
if (response.getStatusCode() == 201) {
|
||||
System.out.println("Blocklist created");
|
||||
} else if (response.getStatusCode() == 200) {
|
||||
System.out.println("Blocklist updated");
|
||||
}
|
||||
```
|
||||
|
||||
### Add Block Items
|
||||
|
||||
```java
|
||||
import com.azure.ai.contentsafety.models.*;
|
||||
import java.util.Arrays;
|
||||
|
||||
List<TextBlocklistItem> items = Arrays.asList(
|
||||
new TextBlocklistItem("badword1").setDescription("Offensive term"),
|
||||
new TextBlocklistItem("badword2").setDescription("Another term")
|
||||
);
|
||||
|
||||
AddOrUpdateTextBlocklistItemsResult result = blocklistClient.addOrUpdateBlocklistItems(
|
||||
"my-blocklist",
|
||||
new AddOrUpdateTextBlocklistItemsOptions(items));
|
||||
|
||||
for (TextBlocklistItem item : result.getBlocklistItems()) {
|
||||
System.out.printf("Added: %s (ID: %s)%n",
|
||||
item.getText(),
|
||||
item.getBlocklistItemId());
|
||||
}
|
||||
```
|
||||
|
||||
### List Blocklists
|
||||
|
||||
```java
|
||||
PagedIterable<TextBlocklist> blocklists = blocklistClient.listTextBlocklists();
|
||||
|
||||
for (TextBlocklist blocklist : blocklists) {
|
||||
System.out.printf("Blocklist: %s, Description: %s%n",
|
||||
blocklist.getName(),
|
||||
blocklist.getDescription());
|
||||
}
|
||||
```
|
||||
|
||||
### Get Blocklist
|
||||
|
||||
```java
|
||||
TextBlocklist blocklist = blocklistClient.getTextBlocklist("my-blocklist");
|
||||
System.out.println("Name: " + blocklist.getName());
|
||||
```
|
||||
|
||||
### List Block Items
|
||||
|
||||
```java
|
||||
PagedIterable<TextBlocklistItem> items =
|
||||
blocklistClient.listTextBlocklistItems("my-blocklist");
|
||||
|
||||
for (TextBlocklistItem item : items) {
|
||||
System.out.printf("ID: %s, Text: %s%n",
|
||||
item.getBlocklistItemId(),
|
||||
item.getText());
|
||||
}
|
||||
```
|
||||
|
||||
### Remove Block Items
|
||||
|
||||
```java
|
||||
List<String> itemIds = Arrays.asList("item-id-1", "item-id-2");
|
||||
|
||||
blocklistClient.removeBlocklistItems(
|
||||
"my-blocklist",
|
||||
new RemoveTextBlocklistItemsOptions(itemIds));
|
||||
```
|
||||
|
||||
### Delete Blocklist
|
||||
|
||||
```java
|
||||
blocklistClient.deleteTextBlocklist("my-blocklist");
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
```java
|
||||
import com.azure.core.exception.HttpResponseException;
|
||||
|
||||
try {
|
||||
contentSafetyClient.analyzeText(new AnalyzeTextOptions("test"));
|
||||
} catch (HttpResponseException e) {
|
||||
System.out.println("Status: " + e.getResponse().getStatusCode());
|
||||
System.out.println("Error: " + e.getMessage());
|
||||
// Common codes: InvalidRequestBody, ResourceNotFound, TooManyRequests
|
||||
}
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
CONTENT_SAFETY_ENDPOINT=https://<resource>.cognitiveservices.azure.com/
|
||||
CONTENT_SAFETY_KEY=<your-api-key>
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Blocklist Delay**: Changes take ~5 minutes to take effect
|
||||
2. **Category Selection**: Only request needed categories to reduce latency
|
||||
3. **Severity Thresholds**: Typically block severity >= 4 for strict moderation
|
||||
4. **Batch Processing**: Process multiple items in parallel for throughput
|
||||
5. **Caching**: Cache blocklist results where appropriate
|
||||
|
||||
## Trigger Phrases
|
||||
|
||||
- "content safety Java"
|
||||
- "content moderation Azure"
|
||||
- "analyze text safety"
|
||||
- "image moderation Java"
|
||||
- "blocklist management"
|
||||
- "hate speech detection"
|
||||
- "harmful content filter"
|
||||
341
skills/official/microsoft/java/foundry/formrecognizer/SKILL.md
Normal file
341
skills/official/microsoft/java/foundry/formrecognizer/SKILL.md
Normal file
@@ -0,0 +1,341 @@
|
||||
---
|
||||
name: azure-ai-formrecognizer-java
|
||||
description: Build document analysis applications with Azure Document Intelligence (Form Recognizer) SDK for Java. Use when extracting text, tables, key-value pairs from documents, receipts, invoices, or building custom document models.
|
||||
package: com.azure:azure-ai-formrecognizer
|
||||
---
|
||||
|
||||
# Azure Document Intelligence (Form Recognizer) SDK for Java
|
||||
|
||||
Build document analysis applications using the Azure AI Document Intelligence SDK for Java.
|
||||
|
||||
## Installation
|
||||
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>com.azure</groupId>
|
||||
<artifactId>azure-ai-formrecognizer</artifactId>
|
||||
<version>4.2.0-beta.1</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
## Client Creation
|
||||
|
||||
### DocumentAnalysisClient
|
||||
|
||||
```java
|
||||
import com.azure.ai.formrecognizer.documentanalysis.DocumentAnalysisClient;
|
||||
import com.azure.ai.formrecognizer.documentanalysis.DocumentAnalysisClientBuilder;
|
||||
import com.azure.core.credential.AzureKeyCredential;
|
||||
|
||||
DocumentAnalysisClient client = new DocumentAnalysisClientBuilder()
|
||||
.credential(new AzureKeyCredential("{key}"))
|
||||
.endpoint("{endpoint}")
|
||||
.buildClient();
|
||||
```
|
||||
|
||||
### DocumentModelAdministrationClient
|
||||
|
||||
```java
|
||||
import com.azure.ai.formrecognizer.documentanalysis.administration.DocumentModelAdministrationClient;
|
||||
import com.azure.ai.formrecognizer.documentanalysis.administration.DocumentModelAdministrationClientBuilder;
|
||||
|
||||
DocumentModelAdministrationClient adminClient = new DocumentModelAdministrationClientBuilder()
|
||||
.credential(new AzureKeyCredential("{key}"))
|
||||
.endpoint("{endpoint}")
|
||||
.buildClient();
|
||||
```
|
||||
|
||||
### With DefaultAzureCredential
|
||||
|
||||
```java
|
||||
import com.azure.identity.DefaultAzureCredentialBuilder;
|
||||
|
||||
DocumentAnalysisClient client = new DocumentAnalysisClientBuilder()
|
||||
.endpoint("{endpoint}")
|
||||
.credential(new DefaultAzureCredentialBuilder().build())
|
||||
.buildClient();
|
||||
```
|
||||
|
||||
## Prebuilt Models
|
||||
|
||||
| Model ID | Purpose |
|
||||
|----------|---------|
|
||||
| `prebuilt-layout` | Extract text, tables, selection marks |
|
||||
| `prebuilt-document` | General document with key-value pairs |
|
||||
| `prebuilt-receipt` | Receipt data extraction |
|
||||
| `prebuilt-invoice` | Invoice field extraction |
|
||||
| `prebuilt-businessCard` | Business card parsing |
|
||||
| `prebuilt-idDocument` | ID document (passport, license) |
|
||||
| `prebuilt-tax.us.w2` | US W2 tax forms |
|
||||
|
||||
## Core Patterns
|
||||
|
||||
### Extract Layout
|
||||
|
||||
```java
|
||||
import com.azure.ai.formrecognizer.documentanalysis.models.*;
|
||||
import com.azure.core.util.BinaryData;
|
||||
import com.azure.core.util.polling.SyncPoller;
|
||||
import java.io.File;
|
||||
|
||||
File document = new File("document.pdf");
|
||||
BinaryData documentData = BinaryData.fromFile(document.toPath());
|
||||
|
||||
SyncPoller<OperationResult, AnalyzeResult> poller =
|
||||
client.beginAnalyzeDocument("prebuilt-layout", documentData);
|
||||
|
||||
AnalyzeResult result = poller.getFinalResult();
|
||||
|
||||
// Process pages
|
||||
for (DocumentPage page : result.getPages()) {
|
||||
System.out.printf("Page %d: %.2f x %.2f %s%n",
|
||||
page.getPageNumber(),
|
||||
page.getWidth(),
|
||||
page.getHeight(),
|
||||
page.getUnit());
|
||||
|
||||
// Lines
|
||||
for (DocumentLine line : page.getLines()) {
|
||||
System.out.println("Line: " + line.getContent());
|
||||
}
|
||||
|
||||
// Selection marks (checkboxes)
|
||||
for (DocumentSelectionMark mark : page.getSelectionMarks()) {
|
||||
System.out.printf("Checkbox: %s (confidence: %.2f)%n",
|
||||
mark.getSelectionMarkState(),
|
||||
mark.getConfidence());
|
||||
}
|
||||
}
|
||||
|
||||
// Tables
|
||||
for (DocumentTable table : result.getTables()) {
|
||||
System.out.printf("Table: %d rows x %d columns%n",
|
||||
table.getRowCount(),
|
||||
table.getColumnCount());
|
||||
|
||||
for (DocumentTableCell cell : table.getCells()) {
|
||||
System.out.printf("Cell[%d,%d]: %s%n",
|
||||
cell.getRowIndex(),
|
||||
cell.getColumnIndex(),
|
||||
cell.getContent());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Analyze from URL
|
||||
|
||||
```java
|
||||
String documentUrl = "https://example.com/invoice.pdf";
|
||||
|
||||
SyncPoller<OperationResult, AnalyzeResult> poller =
|
||||
client.beginAnalyzeDocumentFromUrl("prebuilt-invoice", documentUrl);
|
||||
|
||||
AnalyzeResult result = poller.getFinalResult();
|
||||
```
|
||||
|
||||
### Analyze Receipt
|
||||
|
||||
```java
|
||||
SyncPoller<OperationResult, AnalyzeResult> poller =
|
||||
client.beginAnalyzeDocumentFromUrl("prebuilt-receipt", receiptUrl);
|
||||
|
||||
AnalyzeResult result = poller.getFinalResult();
|
||||
|
||||
for (AnalyzedDocument doc : result.getDocuments()) {
|
||||
Map<String, DocumentField> fields = doc.getFields();
|
||||
|
||||
DocumentField merchantName = fields.get("MerchantName");
|
||||
if (merchantName != null && merchantName.getType() == DocumentFieldType.STRING) {
|
||||
System.out.printf("Merchant: %s (confidence: %.2f)%n",
|
||||
merchantName.getValueAsString(),
|
||||
merchantName.getConfidence());
|
||||
}
|
||||
|
||||
DocumentField transactionDate = fields.get("TransactionDate");
|
||||
if (transactionDate != null && transactionDate.getType() == DocumentFieldType.DATE) {
|
||||
System.out.printf("Date: %s%n", transactionDate.getValueAsDate());
|
||||
}
|
||||
|
||||
DocumentField items = fields.get("Items");
|
||||
if (items != null && items.getType() == DocumentFieldType.LIST) {
|
||||
for (DocumentField item : items.getValueAsList()) {
|
||||
Map<String, DocumentField> itemFields = item.getValueAsMap();
|
||||
System.out.printf("Item: %s, Price: %.2f%n",
|
||||
itemFields.get("Name").getValueAsString(),
|
||||
itemFields.get("Price").getValueAsDouble());
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### General Document Analysis
|
||||
|
||||
```java
|
||||
SyncPoller<OperationResult, AnalyzeResult> poller =
|
||||
client.beginAnalyzeDocumentFromUrl("prebuilt-document", documentUrl);
|
||||
|
||||
AnalyzeResult result = poller.getFinalResult();
|
||||
|
||||
// Key-value pairs
|
||||
for (DocumentKeyValuePair kvp : result.getKeyValuePairs()) {
|
||||
System.out.printf("Key: %s => Value: %s%n",
|
||||
kvp.getKey().getContent(),
|
||||
kvp.getValue() != null ? kvp.getValue().getContent() : "null");
|
||||
}
|
||||
```
|
||||
|
||||
## Custom Models
|
||||
|
||||
### Build Custom Model
|
||||
|
||||
```java
|
||||
import com.azure.ai.formrecognizer.documentanalysis.administration.models.*;
|
||||
|
||||
String blobContainerUrl = "{SAS_URL_of_training_data}";
|
||||
String prefix = "training-docs/";
|
||||
|
||||
SyncPoller<OperationResult, DocumentModelDetails> poller = adminClient.beginBuildDocumentModel(
|
||||
blobContainerUrl,
|
||||
DocumentModelBuildMode.TEMPLATE,
|
||||
prefix,
|
||||
new BuildDocumentModelOptions()
|
||||
.setModelId("my-custom-model")
|
||||
.setDescription("Custom invoice model"),
|
||||
Context.NONE);
|
||||
|
||||
DocumentModelDetails model = poller.getFinalResult();
|
||||
|
||||
System.out.println("Model ID: " + model.getModelId());
|
||||
System.out.println("Created: " + model.getCreatedOn());
|
||||
|
||||
model.getDocumentTypes().forEach((docType, details) -> {
|
||||
System.out.println("Document type: " + docType);
|
||||
details.getFieldSchema().forEach((field, schema) -> {
|
||||
System.out.printf(" Field: %s (%s)%n", field, schema.getType());
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Analyze with Custom Model
|
||||
|
||||
```java
|
||||
SyncPoller<OperationResult, AnalyzeResult> poller =
|
||||
client.beginAnalyzeDocumentFromUrl("my-custom-model", documentUrl);
|
||||
|
||||
AnalyzeResult result = poller.getFinalResult();
|
||||
|
||||
for (AnalyzedDocument doc : result.getDocuments()) {
|
||||
System.out.printf("Document type: %s (confidence: %.2f)%n",
|
||||
doc.getDocType(),
|
||||
doc.getConfidence());
|
||||
|
||||
doc.getFields().forEach((name, field) -> {
|
||||
System.out.printf("Field '%s': %s (confidence: %.2f)%n",
|
||||
name,
|
||||
field.getContent(),
|
||||
field.getConfidence());
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Compose Models
|
||||
|
||||
```java
|
||||
List<String> modelIds = Arrays.asList("model-1", "model-2", "model-3");
|
||||
|
||||
SyncPoller<OperationResult, DocumentModelDetails> poller =
|
||||
adminClient.beginComposeDocumentModel(
|
||||
modelIds,
|
||||
new ComposeDocumentModelOptions()
|
||||
.setModelId("composed-model")
|
||||
.setDescription("Composed from multiple models"));
|
||||
|
||||
DocumentModelDetails composedModel = poller.getFinalResult();
|
||||
```
|
||||
|
||||
### Manage Models
|
||||
|
||||
```java
|
||||
// List models
|
||||
PagedIterable<DocumentModelSummary> models = adminClient.listDocumentModels();
|
||||
for (DocumentModelSummary summary : models) {
|
||||
System.out.printf("Model: %s, Created: %s%n",
|
||||
summary.getModelId(),
|
||||
summary.getCreatedOn());
|
||||
}
|
||||
|
||||
// Get model details
|
||||
DocumentModelDetails model = adminClient.getDocumentModel("model-id");
|
||||
|
||||
// Delete model
|
||||
adminClient.deleteDocumentModel("model-id");
|
||||
|
||||
// Check resource limits
|
||||
ResourceDetails resources = adminClient.getResourceDetails();
|
||||
System.out.printf("Models: %d / %d%n",
|
||||
resources.getCustomDocumentModelCount(),
|
||||
resources.getCustomDocumentModelLimit());
|
||||
```
|
||||
|
||||
## Document Classification
|
||||
|
||||
### Build Classifier
|
||||
|
||||
```java
|
||||
Map<String, ClassifierDocumentTypeDetails> docTypes = new HashMap<>();
|
||||
docTypes.put("invoice", new ClassifierDocumentTypeDetails()
|
||||
.setAzureBlobSource(new AzureBlobContentSource(containerUrl).setPrefix("invoices/")));
|
||||
docTypes.put("receipt", new ClassifierDocumentTypeDetails()
|
||||
.setAzureBlobSource(new AzureBlobContentSource(containerUrl).setPrefix("receipts/")));
|
||||
|
||||
SyncPoller<OperationResult, DocumentClassifierDetails> poller =
|
||||
adminClient.beginBuildDocumentClassifier(docTypes,
|
||||
new BuildDocumentClassifierOptions().setClassifierId("my-classifier"));
|
||||
|
||||
DocumentClassifierDetails classifier = poller.getFinalResult();
|
||||
```
|
||||
|
||||
### Classify Document
|
||||
|
||||
```java
|
||||
SyncPoller<OperationResult, AnalyzeResult> poller =
|
||||
client.beginClassifyDocumentFromUrl("my-classifier", documentUrl, Context.NONE);
|
||||
|
||||
AnalyzeResult result = poller.getFinalResult();
|
||||
|
||||
for (AnalyzedDocument doc : result.getDocuments()) {
|
||||
System.out.printf("Classified as: %s (confidence: %.2f)%n",
|
||||
doc.getDocType(),
|
||||
doc.getConfidence());
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
```java
|
||||
import com.azure.core.exception.HttpResponseException;
|
||||
|
||||
try {
|
||||
client.beginAnalyzeDocumentFromUrl("prebuilt-receipt", "invalid-url");
|
||||
} catch (HttpResponseException e) {
|
||||
System.out.println("Status: " + e.getResponse().getStatusCode());
|
||||
System.out.println("Error: " + e.getMessage());
|
||||
}
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
FORM_RECOGNIZER_ENDPOINT=https://<resource>.cognitiveservices.azure.com/
|
||||
FORM_RECOGNIZER_KEY=<your-api-key>
|
||||
```
|
||||
|
||||
## Trigger Phrases
|
||||
|
||||
- "document intelligence Java"
|
||||
- "form recognizer SDK"
|
||||
- "extract text from PDF"
|
||||
- "OCR document Java"
|
||||
- "analyze invoice receipt"
|
||||
- "custom document model"
|
||||
- "document classification"
|
||||
152
skills/official/microsoft/java/foundry/projects/SKILL.md
Normal file
152
skills/official/microsoft/java/foundry/projects/SKILL.md
Normal file
@@ -0,0 +1,152 @@
|
||||
---
|
||||
name: azure-ai-projects-java
|
||||
description: |
|
||||
Azure AI Projects SDK for Java. High-level SDK for Azure AI Foundry project management including connections, datasets, indexes, and evaluations.
|
||||
Triggers: "AIProjectClient java", "azure ai projects java", "Foundry project java", "ConnectionsClient", "DatasetsClient", "IndexesClient".
|
||||
package: com.azure:azure-ai-projects
|
||||
---
|
||||
|
||||
# Azure AI Projects SDK for Java
|
||||
|
||||
High-level SDK for Azure AI Foundry project management with access to connections, datasets, indexes, and evaluations.
|
||||
|
||||
## Installation
|
||||
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>com.azure</groupId>
|
||||
<artifactId>azure-ai-projects</artifactId>
|
||||
<version>1.0.0-beta.1</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
PROJECT_ENDPOINT=https://<resource>.services.ai.azure.com/api/projects/<project>
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
```java
|
||||
import com.azure.ai.projects.AIProjectClientBuilder;
|
||||
import com.azure.identity.DefaultAzureCredentialBuilder;
|
||||
|
||||
AIProjectClientBuilder builder = new AIProjectClientBuilder()
|
||||
.endpoint(System.getenv("PROJECT_ENDPOINT"))
|
||||
.credential(new DefaultAzureCredentialBuilder().build());
|
||||
```
|
||||
|
||||
## Client Hierarchy
|
||||
|
||||
The SDK provides multiple sub-clients for different operations:
|
||||
|
||||
| Client | Purpose |
|
||||
|--------|---------|
|
||||
| `ConnectionsClient` | Enumerate connected Azure resources |
|
||||
| `DatasetsClient` | Upload documents and manage datasets |
|
||||
| `DeploymentsClient` | Enumerate AI model deployments |
|
||||
| `IndexesClient` | Create and manage search indexes |
|
||||
| `EvaluationsClient` | Run AI model evaluations |
|
||||
| `EvaluatorsClient` | Manage evaluator configurations |
|
||||
| `SchedulesClient` | Manage scheduled operations |
|
||||
|
||||
```java
|
||||
// Build sub-clients from builder
|
||||
ConnectionsClient connectionsClient = builder.buildConnectionsClient();
|
||||
DatasetsClient datasetsClient = builder.buildDatasetsClient();
|
||||
DeploymentsClient deploymentsClient = builder.buildDeploymentsClient();
|
||||
IndexesClient indexesClient = builder.buildIndexesClient();
|
||||
EvaluationsClient evaluationsClient = builder.buildEvaluationsClient();
|
||||
```
|
||||
|
||||
## Core Operations
|
||||
|
||||
### List Connections
|
||||
|
||||
```java
|
||||
import com.azure.ai.projects.models.Connection;
|
||||
import com.azure.core.http.rest.PagedIterable;
|
||||
|
||||
PagedIterable<Connection> connections = connectionsClient.listConnections();
|
||||
for (Connection connection : connections) {
|
||||
System.out.println("Name: " + connection.getName());
|
||||
System.out.println("Type: " + connection.getType());
|
||||
System.out.println("Credential Type: " + connection.getCredentials().getType());
|
||||
}
|
||||
```
|
||||
|
||||
### List Indexes
|
||||
|
||||
```java
|
||||
indexesClient.listLatest().forEach(index -> {
|
||||
System.out.println("Index name: " + index.getName());
|
||||
System.out.println("Version: " + index.getVersion());
|
||||
System.out.println("Description: " + index.getDescription());
|
||||
});
|
||||
```
|
||||
|
||||
### Create or Update Index
|
||||
|
||||
```java
|
||||
import com.azure.ai.projects.models.AzureAISearchIndex;
|
||||
import com.azure.ai.projects.models.Index;
|
||||
|
||||
String indexName = "my-index";
|
||||
String indexVersion = "1.0";
|
||||
String searchConnectionName = System.getenv("AI_SEARCH_CONNECTION_NAME");
|
||||
String searchIndexName = System.getenv("AI_SEARCH_INDEX_NAME");
|
||||
|
||||
Index index = indexesClient.createOrUpdate(
|
||||
indexName,
|
||||
indexVersion,
|
||||
new AzureAISearchIndex()
|
||||
.setConnectionName(searchConnectionName)
|
||||
.setIndexName(searchIndexName)
|
||||
);
|
||||
|
||||
System.out.println("Created index: " + index.getName());
|
||||
```
|
||||
|
||||
### Access OpenAI Evaluations
|
||||
|
||||
The SDK exposes OpenAI's official SDK for evaluations:
|
||||
|
||||
```java
|
||||
import com.openai.services.EvalService;
|
||||
|
||||
EvalService evalService = evaluationsClient.getOpenAIClient();
|
||||
// Use OpenAI evaluation APIs directly
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use DefaultAzureCredential** for production authentication
|
||||
2. **Reuse client builder** to create multiple sub-clients efficiently
|
||||
3. **Handle pagination** when listing resources with `PagedIterable`
|
||||
4. **Use environment variables** for connection names and configuration
|
||||
5. **Check connection types** before accessing credentials
|
||||
|
||||
## Error Handling
|
||||
|
||||
```java
|
||||
import com.azure.core.exception.HttpResponseException;
|
||||
import com.azure.core.exception.ResourceNotFoundException;
|
||||
|
||||
try {
|
||||
Index index = indexesClient.get(indexName, version);
|
||||
} catch (ResourceNotFoundException e) {
|
||||
System.err.println("Index not found: " + indexName);
|
||||
} catch (HttpResponseException e) {
|
||||
System.err.println("Error: " + e.getResponse().getStatusCode());
|
||||
}
|
||||
```
|
||||
|
||||
## Reference Links
|
||||
|
||||
| Resource | URL |
|
||||
|----------|-----|
|
||||
| Product Docs | https://learn.microsoft.com/azure/ai-studio/ |
|
||||
| API Reference | https://learn.microsoft.com/rest/api/aifoundry/aiprojects/ |
|
||||
| GitHub Source | https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/ai/azure-ai-projects |
|
||||
| Samples | https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/ai/azure-ai-projects/src/samples |
|
||||
@@ -0,0 +1,289 @@
|
||||
---
|
||||
name: azure-ai-vision-imageanalysis-java
|
||||
description: Build image analysis applications with Azure AI Vision SDK for Java. Use when implementing image captioning, OCR text extraction, object detection, tagging, or smart cropping.
|
||||
package: com.azure:azure-ai-vision-imageanalysis
|
||||
---
|
||||
|
||||
# Azure AI Vision Image Analysis SDK for Java
|
||||
|
||||
Build image analysis applications using the Azure AI Vision Image Analysis SDK for Java.
|
||||
|
||||
## Installation
|
||||
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>com.azure</groupId>
|
||||
<artifactId>azure-ai-vision-imageanalysis</artifactId>
|
||||
<version>1.1.0-beta.1</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
## Client Creation
|
||||
|
||||
### With API Key
|
||||
|
||||
```java
|
||||
import com.azure.ai.vision.imageanalysis.ImageAnalysisClient;
|
||||
import com.azure.ai.vision.imageanalysis.ImageAnalysisClientBuilder;
|
||||
import com.azure.core.credential.KeyCredential;
|
||||
|
||||
String endpoint = System.getenv("VISION_ENDPOINT");
|
||||
String key = System.getenv("VISION_KEY");
|
||||
|
||||
ImageAnalysisClient client = new ImageAnalysisClientBuilder()
|
||||
.endpoint(endpoint)
|
||||
.credential(new KeyCredential(key))
|
||||
.buildClient();
|
||||
```
|
||||
|
||||
### Async Client
|
||||
|
||||
```java
|
||||
import com.azure.ai.vision.imageanalysis.ImageAnalysisAsyncClient;
|
||||
|
||||
ImageAnalysisAsyncClient asyncClient = new ImageAnalysisClientBuilder()
|
||||
.endpoint(endpoint)
|
||||
.credential(new KeyCredential(key))
|
||||
.buildAsyncClient();
|
||||
```
|
||||
|
||||
### With DefaultAzureCredential
|
||||
|
||||
```java
|
||||
import com.azure.identity.DefaultAzureCredentialBuilder;
|
||||
|
||||
ImageAnalysisClient client = new ImageAnalysisClientBuilder()
|
||||
.endpoint(endpoint)
|
||||
.credential(new DefaultAzureCredentialBuilder().build())
|
||||
.buildClient();
|
||||
```
|
||||
|
||||
## Visual Features
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| `CAPTION` | Generate human-readable image description |
|
||||
| `DENSE_CAPTIONS` | Captions for up to 10 regions |
|
||||
| `READ` | OCR - Extract text from images |
|
||||
| `TAGS` | Content tags for objects, scenes, actions |
|
||||
| `OBJECTS` | Detect objects with bounding boxes |
|
||||
| `SMART_CROPS` | Smart thumbnail regions |
|
||||
| `PEOPLE` | Detect people with locations |
|
||||
|
||||
## Core Patterns
|
||||
|
||||
### Generate Caption
|
||||
|
||||
```java
|
||||
import com.azure.ai.vision.imageanalysis.models.*;
|
||||
import com.azure.core.util.BinaryData;
|
||||
import java.io.File;
|
||||
import java.util.Arrays;
|
||||
|
||||
// From file
|
||||
BinaryData imageData = BinaryData.fromFile(new File("image.jpg").toPath());
|
||||
|
||||
ImageAnalysisResult result = client.analyze(
|
||||
imageData,
|
||||
Arrays.asList(VisualFeatures.CAPTION),
|
||||
new ImageAnalysisOptions().setGenderNeutralCaption(true));
|
||||
|
||||
System.out.printf("Caption: \"%s\" (confidence: %.4f)%n",
|
||||
result.getCaption().getText(),
|
||||
result.getCaption().getConfidence());
|
||||
```
|
||||
|
||||
### Generate Caption from URL
|
||||
|
||||
```java
|
||||
ImageAnalysisResult result = client.analyzeFromUrl(
|
||||
"https://example.com/image.jpg",
|
||||
Arrays.asList(VisualFeatures.CAPTION),
|
||||
new ImageAnalysisOptions().setGenderNeutralCaption(true));
|
||||
|
||||
System.out.printf("Caption: \"%s\"%n", result.getCaption().getText());
|
||||
```
|
||||
|
||||
### Extract Text (OCR)
|
||||
|
||||
```java
|
||||
ImageAnalysisResult result = client.analyze(
|
||||
BinaryData.fromFile(new File("document.jpg").toPath()),
|
||||
Arrays.asList(VisualFeatures.READ),
|
||||
null);
|
||||
|
||||
for (DetectedTextBlock block : result.getRead().getBlocks()) {
|
||||
for (DetectedTextLine line : block.getLines()) {
|
||||
System.out.printf("Line: '%s'%n", line.getText());
|
||||
System.out.printf(" Bounding polygon: %s%n", line.getBoundingPolygon());
|
||||
|
||||
for (DetectedTextWord word : line.getWords()) {
|
||||
System.out.printf(" Word: '%s' (confidence: %.4f)%n",
|
||||
word.getText(),
|
||||
word.getConfidence());
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Detect Objects
|
||||
|
||||
```java
|
||||
ImageAnalysisResult result = client.analyzeFromUrl(
|
||||
imageUrl,
|
||||
Arrays.asList(VisualFeatures.OBJECTS),
|
||||
null);
|
||||
|
||||
for (DetectedObject obj : result.getObjects()) {
|
||||
System.out.printf("Object: %s (confidence: %.4f)%n",
|
||||
obj.getTags().get(0).getName(),
|
||||
obj.getTags().get(0).getConfidence());
|
||||
|
||||
ImageBoundingBox box = obj.getBoundingBox();
|
||||
System.out.printf(" Location: x=%d, y=%d, w=%d, h=%d%n",
|
||||
box.getX(), box.getY(), box.getWidth(), box.getHeight());
|
||||
}
|
||||
```
|
||||
|
||||
### Get Tags
|
||||
|
||||
```java
|
||||
ImageAnalysisResult result = client.analyzeFromUrl(
|
||||
imageUrl,
|
||||
Arrays.asList(VisualFeatures.TAGS),
|
||||
null);
|
||||
|
||||
for (DetectedTag tag : result.getTags()) {
|
||||
System.out.printf("Tag: %s (confidence: %.4f)%n",
|
||||
tag.getName(),
|
||||
tag.getConfidence());
|
||||
}
|
||||
```
|
||||
|
||||
### Detect People
|
||||
|
||||
```java
|
||||
ImageAnalysisResult result = client.analyzeFromUrl(
|
||||
imageUrl,
|
||||
Arrays.asList(VisualFeatures.PEOPLE),
|
||||
null);
|
||||
|
||||
for (DetectedPerson person : result.getPeople()) {
|
||||
ImageBoundingBox box = person.getBoundingBox();
|
||||
System.out.printf("Person at x=%d, y=%d (confidence: %.4f)%n",
|
||||
box.getX(), box.getY(), person.getConfidence());
|
||||
}
|
||||
```
|
||||
|
||||
### Smart Cropping
|
||||
|
||||
```java
|
||||
ImageAnalysisResult result = client.analyzeFromUrl(
|
||||
imageUrl,
|
||||
Arrays.asList(VisualFeatures.SMART_CROPS),
|
||||
new ImageAnalysisOptions().setSmartCropsAspectRatios(Arrays.asList(1.0, 1.5)));
|
||||
|
||||
for (CropRegion crop : result.getSmartCrops()) {
|
||||
System.out.printf("Crop region: aspect=%.2f, x=%d, y=%d, w=%d, h=%d%n",
|
||||
crop.getAspectRatio(),
|
||||
crop.getBoundingBox().getX(),
|
||||
crop.getBoundingBox().getY(),
|
||||
crop.getBoundingBox().getWidth(),
|
||||
crop.getBoundingBox().getHeight());
|
||||
}
|
||||
```
|
||||
|
||||
### Dense Captions
|
||||
|
||||
```java
|
||||
ImageAnalysisResult result = client.analyzeFromUrl(
|
||||
imageUrl,
|
||||
Arrays.asList(VisualFeatures.DENSE_CAPTIONS),
|
||||
new ImageAnalysisOptions().setGenderNeutralCaption(true));
|
||||
|
||||
for (DenseCaption caption : result.getDenseCaptions()) {
|
||||
System.out.printf("Caption: \"%s\" (confidence: %.4f)%n",
|
||||
caption.getText(),
|
||||
caption.getConfidence());
|
||||
System.out.printf(" Region: x=%d, y=%d, w=%d, h=%d%n",
|
||||
caption.getBoundingBox().getX(),
|
||||
caption.getBoundingBox().getY(),
|
||||
caption.getBoundingBox().getWidth(),
|
||||
caption.getBoundingBox().getHeight());
|
||||
}
|
||||
```
|
||||
|
||||
### Multiple Features
|
||||
|
||||
```java
|
||||
ImageAnalysisResult result = client.analyzeFromUrl(
|
||||
imageUrl,
|
||||
Arrays.asList(
|
||||
VisualFeatures.CAPTION,
|
||||
VisualFeatures.TAGS,
|
||||
VisualFeatures.OBJECTS,
|
||||
VisualFeatures.READ),
|
||||
new ImageAnalysisOptions()
|
||||
.setGenderNeutralCaption(true)
|
||||
.setLanguage("en"));
|
||||
|
||||
// Access all results
|
||||
System.out.println("Caption: " + result.getCaption().getText());
|
||||
System.out.println("Tags: " + result.getTags().size());
|
||||
System.out.println("Objects: " + result.getObjects().size());
|
||||
System.out.println("Text blocks: " + result.getRead().getBlocks().size());
|
||||
```
|
||||
|
||||
### Async Analysis
|
||||
|
||||
```java
|
||||
asyncClient.analyzeFromUrl(
|
||||
imageUrl,
|
||||
Arrays.asList(VisualFeatures.CAPTION),
|
||||
null)
|
||||
.subscribe(
|
||||
result -> System.out.println("Caption: " + result.getCaption().getText()),
|
||||
error -> System.err.println("Error: " + error.getMessage()),
|
||||
() -> System.out.println("Complete")
|
||||
);
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
```java
|
||||
import com.azure.core.exception.HttpResponseException;
|
||||
|
||||
try {
|
||||
client.analyzeFromUrl(imageUrl, Arrays.asList(VisualFeatures.CAPTION), null);
|
||||
} catch (HttpResponseException e) {
|
||||
System.out.println("Status: " + e.getResponse().getStatusCode());
|
||||
System.out.println("Error: " + e.getMessage());
|
||||
}
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
VISION_ENDPOINT=https://<resource>.cognitiveservices.azure.com/
|
||||
VISION_KEY=<your-api-key>
|
||||
```
|
||||
|
||||
## Image Requirements
|
||||
|
||||
- Formats: JPEG, PNG, GIF, BMP, WEBP, ICO, TIFF, MPO
|
||||
- Size: < 20 MB
|
||||
- Dimensions: 50x50 to 16000x16000 pixels
|
||||
|
||||
## Regional Availability
|
||||
|
||||
Caption and Dense Captions require GPU-supported regions. Check [supported regions](https://learn.microsoft.com/azure/ai-services/computer-vision/concept-describe-images-40) before deployment.
|
||||
|
||||
## Trigger Phrases
|
||||
|
||||
- "image analysis Java"
|
||||
- "Azure Vision SDK"
|
||||
- "image captioning"
|
||||
- "OCR image text extraction"
|
||||
- "object detection image"
|
||||
- "smart crop thumbnail"
|
||||
- "detect people image"
|
||||
225
skills/official/microsoft/java/foundry/voicelive/SKILL.md
Normal file
225
skills/official/microsoft/java/foundry/voicelive/SKILL.md
Normal file
@@ -0,0 +1,225 @@
|
||||
---
|
||||
name: azure-ai-voicelive-java
|
||||
description: |
|
||||
Azure AI VoiceLive SDK for Java. Real-time bidirectional voice conversations with AI assistants using WebSocket.
|
||||
Triggers: "VoiceLiveClient java", "voice assistant java", "real-time voice java", "audio streaming java", "voice activity detection java".
|
||||
package: com.azure:azure-ai-voicelive
|
||||
---
|
||||
|
||||
# Azure AI VoiceLive SDK for Java
|
||||
|
||||
Real-time, bidirectional voice conversations with AI assistants using WebSocket technology.
|
||||
|
||||
## Installation
|
||||
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>com.azure</groupId>
|
||||
<artifactId>azure-ai-voicelive</artifactId>
|
||||
<version>1.0.0-beta.2</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
AZURE_VOICELIVE_ENDPOINT=https://<resource>.openai.azure.com/
|
||||
AZURE_VOICELIVE_API_KEY=<your-api-key>
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
### API Key
|
||||
|
||||
```java
|
||||
import com.azure.ai.voicelive.VoiceLiveAsyncClient;
|
||||
import com.azure.ai.voicelive.VoiceLiveClientBuilder;
|
||||
import com.azure.core.credential.AzureKeyCredential;
|
||||
|
||||
VoiceLiveAsyncClient client = new VoiceLiveClientBuilder()
|
||||
.endpoint(System.getenv("AZURE_VOICELIVE_ENDPOINT"))
|
||||
.credential(new AzureKeyCredential(System.getenv("AZURE_VOICELIVE_API_KEY")))
|
||||
.buildAsyncClient();
|
||||
```
|
||||
|
||||
### DefaultAzureCredential (Recommended)
|
||||
|
||||
```java
|
||||
import com.azure.identity.DefaultAzureCredentialBuilder;
|
||||
|
||||
VoiceLiveAsyncClient client = new VoiceLiveClientBuilder()
|
||||
.endpoint(System.getenv("AZURE_VOICELIVE_ENDPOINT"))
|
||||
.credential(new DefaultAzureCredentialBuilder().build())
|
||||
.buildAsyncClient();
|
||||
```
|
||||
|
||||
## Key Concepts
|
||||
|
||||
| Concept | Description |
|
||||
|---------|-------------|
|
||||
| `VoiceLiveAsyncClient` | Main entry point for voice sessions |
|
||||
| `VoiceLiveSessionAsyncClient` | Active WebSocket connection for streaming |
|
||||
| `VoiceLiveSessionOptions` | Configuration for session behavior |
|
||||
|
||||
### Audio Requirements
|
||||
|
||||
- **Sample Rate**: 24kHz (24000 Hz)
|
||||
- **Bit Depth**: 16-bit PCM
|
||||
- **Channels**: Mono (1 channel)
|
||||
- **Format**: Signed PCM, little-endian
|
||||
|
||||
## Core Workflow
|
||||
|
||||
### 1. Start Session
|
||||
|
||||
```java
|
||||
import reactor.core.publisher.Mono;
|
||||
|
||||
client.startSession("gpt-4o-realtime-preview")
|
||||
.flatMap(session -> {
|
||||
System.out.println("Session started");
|
||||
|
||||
// Subscribe to events
|
||||
session.receiveEvents()
|
||||
.subscribe(
|
||||
event -> System.out.println("Event: " + event.getType()),
|
||||
error -> System.err.println("Error: " + error.getMessage())
|
||||
);
|
||||
|
||||
return Mono.just(session);
|
||||
})
|
||||
.block();
|
||||
```
|
||||
|
||||
### 2. Configure Session Options
|
||||
|
||||
```java
|
||||
import com.azure.ai.voicelive.models.*;
|
||||
import java.util.Arrays;
|
||||
|
||||
ServerVadTurnDetection turnDetection = new ServerVadTurnDetection()
|
||||
.setThreshold(0.5) // Sensitivity (0.0-1.0)
|
||||
.setPrefixPaddingMs(300) // Audio before speech
|
||||
.setSilenceDurationMs(500) // Silence to end turn
|
||||
.setInterruptResponse(true) // Allow interruptions
|
||||
.setAutoTruncate(true)
|
||||
.setCreateResponse(true);
|
||||
|
||||
AudioInputTranscriptionOptions transcription = new AudioInputTranscriptionOptions(
|
||||
AudioInputTranscriptionOptionsModel.WHISPER_1);
|
||||
|
||||
VoiceLiveSessionOptions options = new VoiceLiveSessionOptions()
|
||||
.setInstructions("You are a helpful AI voice assistant.")
|
||||
.setVoice(BinaryData.fromObject(new OpenAIVoice(OpenAIVoiceName.ALLOY)))
|
||||
.setModalities(Arrays.asList(InteractionModality.TEXT, InteractionModality.AUDIO))
|
||||
.setInputAudioFormat(InputAudioFormat.PCM16)
|
||||
.setOutputAudioFormat(OutputAudioFormat.PCM16)
|
||||
.setInputAudioSamplingRate(24000)
|
||||
.setInputAudioNoiseReduction(new AudioNoiseReduction(AudioNoiseReductionType.NEAR_FIELD))
|
||||
.setInputAudioEchoCancellation(new AudioEchoCancellation())
|
||||
.setInputAudioTranscription(transcription)
|
||||
.setTurnDetection(turnDetection);
|
||||
|
||||
// Send configuration
|
||||
ClientEventSessionUpdate updateEvent = new ClientEventSessionUpdate(options);
|
||||
session.sendEvent(updateEvent).subscribe();
|
||||
```
|
||||
|
||||
### 3. Send Audio Input
|
||||
|
||||
```java
|
||||
byte[] audioData = readAudioChunk(); // Your PCM16 audio data
|
||||
session.sendInputAudio(BinaryData.fromBytes(audioData)).subscribe();
|
||||
```
|
||||
|
||||
### 4. Handle Events
|
||||
|
||||
```java
|
||||
session.receiveEvents().subscribe(event -> {
|
||||
ServerEventType eventType = event.getType();
|
||||
|
||||
if (ServerEventType.SESSION_CREATED.equals(eventType)) {
|
||||
System.out.println("Session created");
|
||||
} else if (ServerEventType.INPUT_AUDIO_BUFFER_SPEECH_STARTED.equals(eventType)) {
|
||||
System.out.println("User started speaking");
|
||||
} else if (ServerEventType.INPUT_AUDIO_BUFFER_SPEECH_STOPPED.equals(eventType)) {
|
||||
System.out.println("User stopped speaking");
|
||||
} else if (ServerEventType.RESPONSE_AUDIO_DELTA.equals(eventType)) {
|
||||
if (event instanceof SessionUpdateResponseAudioDelta) {
|
||||
SessionUpdateResponseAudioDelta audioEvent = (SessionUpdateResponseAudioDelta) event;
|
||||
playAudioChunk(audioEvent.getDelta());
|
||||
}
|
||||
} else if (ServerEventType.RESPONSE_DONE.equals(eventType)) {
|
||||
System.out.println("Response complete");
|
||||
} else if (ServerEventType.ERROR.equals(eventType)) {
|
||||
if (event instanceof SessionUpdateError) {
|
||||
SessionUpdateError errorEvent = (SessionUpdateError) event;
|
||||
System.err.println("Error: " + errorEvent.getError().getMessage());
|
||||
}
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
## Voice Configuration
|
||||
|
||||
### OpenAI Voices
|
||||
|
||||
```java
|
||||
// Available: ALLOY, ASH, BALLAD, CORAL, ECHO, SAGE, SHIMMER, VERSE
|
||||
VoiceLiveSessionOptions options = new VoiceLiveSessionOptions()
|
||||
.setVoice(BinaryData.fromObject(new OpenAIVoice(OpenAIVoiceName.ALLOY)));
|
||||
```
|
||||
|
||||
### Azure Voices
|
||||
|
||||
```java
|
||||
// Azure Standard Voice
|
||||
options.setVoice(BinaryData.fromObject(new AzureStandardVoice("en-US-JennyNeural")));
|
||||
|
||||
// Azure Custom Voice
|
||||
options.setVoice(BinaryData.fromObject(new AzureCustomVoice("myVoice", "endpointId")));
|
||||
|
||||
// Azure Personal Voice
|
||||
options.setVoice(BinaryData.fromObject(
|
||||
new AzurePersonalVoice("speakerProfileId", PersonalVoiceModels.PHOENIX_LATEST_NEURAL)));
|
||||
```
|
||||
|
||||
## Function Calling
|
||||
|
||||
```java
|
||||
VoiceLiveFunctionDefinition weatherFunction = new VoiceLiveFunctionDefinition("get_weather")
|
||||
.setDescription("Get current weather for a location")
|
||||
.setParameters(BinaryData.fromObject(parametersSchema));
|
||||
|
||||
VoiceLiveSessionOptions options = new VoiceLiveSessionOptions()
|
||||
.setTools(Arrays.asList(weatherFunction))
|
||||
.setInstructions("You have access to weather information.");
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use async client** — VoiceLive requires reactive patterns
|
||||
2. **Configure turn detection** for natural conversation flow
|
||||
3. **Enable noise reduction** for better speech recognition
|
||||
4. **Handle interruptions** gracefully with `setInterruptResponse(true)`
|
||||
5. **Use Whisper transcription** for input audio transcription
|
||||
6. **Close sessions** properly when conversation ends
|
||||
|
||||
## Error Handling
|
||||
|
||||
```java
|
||||
session.receiveEvents()
|
||||
.doOnError(error -> System.err.println("Connection error: " + error.getMessage()))
|
||||
.onErrorResume(error -> {
|
||||
// Attempt reconnection or cleanup
|
||||
return Flux.empty();
|
||||
})
|
||||
.subscribe();
|
||||
```
|
||||
|
||||
## Reference Links
|
||||
|
||||
| Resource | URL |
|
||||
|----------|-----|
|
||||
| GitHub Source | https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/ai/azure-ai-voicelive |
|
||||
| Samples | https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/ai/azure-ai-voicelive/src/samples |
|
||||
Reference in New Issue
Block a user