Llama-3 on Inferentia generate infinite and meaningless output

[yubingjiaocn]

Description

When running Llama-3 on Inferentia lab, inference can be done, but model generate meaningless output after actual content.

• ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Before you submit an issue, please perform the following for Terraform examples:

1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• Module version [Required]: 5a2d1df

• Terraform version:

Terraform v1.8.4
on linux_amd64

• Provider version(s):

+ provider registry.terraform.io/alekc/kubectl v2.0.4
+ provider registry.terraform.io/hashicorp/aws v5.50.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.4
+ provider registry.terraform.io/hashicorp/helm v2.13.2
+ provider registry.terraform.io/hashicorp/http v3.4.2
+ provider registry.terraform.io/hashicorp/kubernetes v2.30.0
+ provider registry.terraform.io/hashicorp/random v3.6.2
+ provider registry.terraform.io/hashicorp/time v0.11.1
+ provider registry.terraform.io/hashicorp/tls v4.0.5

Reproduction Code [Required]

Steps to reproduce the behavior:

1. Follow instruction of the lab to deploy the environment.
2. Use API to inference: accessing http://<nginx address>/serve/infer?sentence=What%20is%20AWS

Expected behavior

Generate an introduction of AWS, and the output only contain the introduction text.

Actual behavior

Generate output based on wrong question, and generate meaningless hashtags after actual answer.

Terminal Output Screenshot(s)

["What is AWS Lambda?\nAWS Lambda is a serverless compute service offered by Amazon Web Services (AWS). It allows you to run code without provisioning or managing servers. You can write and deploy code in a variety of programming languages, including Node.js, Python, Java, Go, and C#.\n\nHere are some key features of AWS Lambda:\n\n1. **Serverless**: You don't need to provision or manage servers. AWS Lambda automatically scales to handle changes in workload.\n2. **Event-driven**: Your code is triggered by events such as changes to an Amazon S3 bucket, Amazon DynamoDB table, or Amazon SQS queue.\n3. **Stateless**: Each invocation of your code is stateless, meaning that it has no knowledge of previous invocations.\n4. **Scalable**: AWS Lambda automatically scales to handle changes in workload, so you don't need to worry about provisioning or managing servers.\n5. **Cost-effective**: You only pay for the compute time consumed by your code, which can be more cost-effective than running a dedicated server.\n6. **Support for multiple programming languages**: You can write and deploy code in a variety of programming languages, including Node.js, Python, Java, Go, and C#.\n7. **Integration with other AWS services**: AWS Lambda integrates with other AWS services, such as Amazon S3, Amazon DynamoDB, Amazon SQS, and Amazon API Gateway.\n\nUse cases for AWS Lambda include:\n\n1. **Real-time data processing**: Use AWS Lambda to process real-time data from sources such as IoT devices, social media, or log files.\n2. **Serverless APIs**: Use AWS Lambda to create serverless APIs that can handle high traffic and scale automatically.\n3. **Background tasks**: Use AWS Lambda to run background tasks, such as sending emails or processing large datasets.\n4. **Machine learning**: Use AWS Lambda to run machine learning models and predict outcomes in real-time.\n\nOverall, AWS Lambda is a powerful tool for building scalable, cost-effective, and highly available applications. It allows you to focus on writing code without worrying about the underlying infrastructure. #AWS #Lambda #Serverless #CloudComputing #DevOps #IT #Technology #Innovation #SoftwareDevelopment #ProgrammingLanguages #MachineLearning #ArtificialIntelligence #DataScience #BigData #CloudNative #CloudComputing #ServerlessComputing #EventDriven #Stateless #Scalable #CostEffective #Integration #APIGateway #S3 #DynamoDB #SQS #MachineLearning #PredictiveAnalytics #RealTimeDataProcessing #BackgroundTasks #ServerlessAPIs #IoT #SocialMedia #LogFiles #DataProcessing #DataAnalytics #DataScience #MachineLearning #ArtificialIntelligence #CloudComputing #ServerlessComputing #EventDriven #Stateless #Scalable #CostEffective #Integration #APIGateway #S3 #DynamoDB #SQS #MachineLearning #PredictiveAnalytics #RealTimeDataProcessing #BackgroundTasks #ServerlessAPIs #IoT #SocialMedia #LogFiles #DataProcessing #DataAnalytics #DataScience #MachineLearning #ArtificialIntelligence #CloudComputing #ServerlessComputing #EventDriven #Stateless #Scalable #CostEffective #Integration #APIGateway #S3 #DynamoDB #SQS #MachineLearning #PredictiveAnalytics #RealTimeDataProcessing #BackgroundTasks #ServerlessAPIs #IoT #SocialMedia #LogFiles #DataProcessing #DataAnalytics #DataScience #MachineLearning #ArtificialIntelligence #CloudComputing #ServerlessComputing #EventDriven #Stateless #Scalable #CostEffective #Integration #APIGateway #S3 #DynamoDB #SQS #MachineLearning #PredictiveAnalytics #RealTimeDataProcessing #BackgroundTasks #ServerlessAPIs #IoT #SocialMedia #LogFiles #DataProcessing #DataAnalytics #DataScience #MachineLearning #ArtificialIntelligence #CloudComputing #ServerlessComputing #EventDriven #Stateless #Scalable #CostEffective #Integration #APIGateway #S3 #DynamoDB #SQS #MachineLearning #PredictiveAnalytics #RealTimeDataProcessing #BackgroundTasks #ServerlessAPIs #IoT #SocialMedia #LogFiles #DataProcessing #DataAnalytics #DataScience #MachineLearning #ArtificialIntelligence #CloudComputing #ServerlessComputing #EventDriven #Stateless #Scalable #CostEffective #Integration #APIGateway #S3 #DynamoDB #SQS #MachineLearning #PredictiveAnalytics #RealTimeDataProcessing #BackgroundTasks #ServerlessAPIs #IoT #SocialMedia #LogFiles #DataProcessing #DataAnalytics #DataScience #MachineLearning #ArtificialIntelligence #CloudComputing #ServerlessComputing #EventDriven #Stateless #Scalable #CostEffective #Integration #APIGateway #S3"]

Additional context

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions]

Issue closed due to inactivity.