Updating saas

[JaladJhamb]

I am actually trying to update saas to v1.1.1, but it shows the following error while launching saas-boost installer:

[PoeppingT]

Hey @JaladJhamb , welcome! Can you provide a little more detail:

1. How did you install SaaS Boost? Do you have git installed locally?
2. Did this error cause your installer to fail? If so, what error was printed out?

[JaladJhamb]

hello @PoeppingT, yes I have got git installed locally and I was actually trying to update it as we are already using, so I am actually trying to update it to version 1.1.1 but as soon as I start updating it, it shows the above error and then comes back to the earlier version.

[JaladJhamb]

Hello @PoeppingT,
I am currently trying to install the SAAS latest version but the cloudformation shows an error in InvokeRDSoptions and the error is:
Received response status [FAILED] from custom resource. Message returned: java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.base/java.util.concurrent.FutureTask.report(Unknown Source) at java.base/java.util.concurrent.FutureTask.get(Unknown Source) at com.amazon.aws.partners.saasfactory.saasboost.RdsOptions.handleRequest(RdsOptions.java:204) at com.amazon.aws.partners.saasfactory.saasboost.RdsOptions.handleRequest(RdsOptions.java:40) at lambdainternal.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:199) at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:899) at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:268) at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:206) at lambdainternal.AWSLambda.main(AWSLambda.java:200) Caused by: java.lang.NullPointerException at com.amazon.aws.partners.saasfactory.saasboost.RdsOptions.lambda$handleRequest$1(RdsOptions.java:131) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) (RequestId: 6b053cbe-94c8-42d6-af32-f8aabf3eb68a)

[PoeppingT]

Hi @JaladJhamb , this looks similar to an issue we have fixed in e2fbf17 . Do you have that commit in your local repository?

You will get better error messages in this case by viewing the CloudWatch LogGroup for the RdsOptions Lambda, named /aws/lambda/sb-${environment}-rds-options. To view these:

1. Go to your AWS Console
2. Navigate to CloudWatch using the Services pane
3. On the left, under Logs, click LogGroups
4. Find the LogGroup named /aws/lambda/sb-${environment}-rds-options by searching in the search box for looking through pages. Note, your environment name should replace ${environment}. So for example if your environment is named main then the full name of the LogGroup will be /aws/lambda/sb-main-rds-options.
5. Scrolling down to Log Streams you should see a link timestamped for when you tried to run the installation. Opening that link will show you the logs.

If you have the above linked commit in your local repository (e2fbf17) and still see an error message, please post it here in its entirety.

[JaladJhamb]

Thank You @PoeppingT,
I want to ask one more thing as I am new into this saas thing, In our QA environment we already have saas deployed but its an older version,
and its causing errors while we are updating it to a new version, so we are actually deploying a new saas of latest version, just want to ask that deploying new version will not have any effect on the older saas which is already deployed or it will cause problems.
Thanks

[PoeppingT]

Hi @JaladJhamb , installing a new environment should have no negative effects on any previous environments you have installed. However, I recommend that if you are not using old environments that you delete them, since any running AWS resources will cost you money over time.

[JaladJhamb]

Thank you for the reply @PoeppingT.
And I actually want to share one thing as I was testing installing latest SAAS, it got install successful and the onboarding of the tenants got in active status in my QA environment and an account called sandbox which we use for testing, but while I am trying to onboard the application in my dev environment it doesn't update any status, it is still in validating status. What might be the reason behind it? as it gets successful in QA and sandbox but onboarding the application on dev doesn't gets into active status, it still in validating status.

[JaladJhamb]

@PoeppingT
This is the screenshot

And as we onboard the application it creates a CloudFormation stack, but in this it is not even creating any so that is why I am unable to troubleshoot the issue.

[PoeppingT]

@JaladJhamb

Before entering the Provisioning state the Onboarding request first has to pass through the Validating state, executed by the sb-${Environment}-onboarding-validation Lambda function. You can view its logs in CloudWatch Logs, the LogGroup will be named /aws/lambda/sb-${Environment}-onboarding-validation.

Validation checks a number of prerequisites that must be met before proceeding with the incoming Onboarding request: AWS quotas, networking requirements, request validity, and the presence of a Docker image in the connected ECR repository. Any of these validation may be failing, and the logs will show which one.

[JaladJhamb]

Thank you so much again @PoeppingT for replying.
I found this in the logs:

begin 0, end -1, length 15: java.lang.StringIndexOutOfBoundsExceptionjava.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 15 at java.base/java.lang.String.checkBoundsBeginEnd(Unknown Source) at java.base/java.lang.String.substring(Unknown Source) at com.amazon.aws.partners.saasfactory.saasboost.OnboardingService.processValidateOnboardingQueue(OnboardingService.java:1437) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source)

begin 0, end -1, length 15: java.lang.StringIndexOutOfBoundsException java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 15 at java.base/java.lang.String.checkBoundsBeginEnd(Unknown Source) at java.base/java.lang.String.substring(Unknown Source) at com.amazon.aws.partners.saasfactory.saasboost.OnboardingService.processValidateOnboardingQueue(OnboardingService.java:1437) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source)

[JaladJhamb]

@PoeppingT please let me know if you have any idea about this error as it is related to production in my project, which needs to be sorted as soon as possible.

[PoeppingT]

@JaladJhamb This error comes from the part of the Onboarding validation code that checks that there is no existing subdomain that matches the one you've requested for your tenant. According to the error location it looks like you have an A record in Route53 that is your domain name. Is that true?

The proper fix would be to replace this A record with an NS record to forward any traffic to an internally managed set of Name Servers, but SaaS Boost can cover for this DNS configuration with some changes to the same Onboarding validation code.

Relevant code from services/onboarding-service/src/main/java/com/amazon/aws/partners/saasfactory/saasboost/OnboardingService.java is below. The .substring is what was failing. Depending on your DNS Record setup in Route53, that probably needs to change to consider the possibility that the A record is the domain name.

ListResourceRecordSetsResponse recordSets = route53.listResourceRecordSets(r -> r
    .hostedZoneId(hostedZoneId)
);
if (recordSets.hasResourceRecordSets()) {
  boolean duplicateSubdomain = false;
  for (ResourceRecordSet recordSet : recordSets.resourceRecordSets()) {
    if (RRType.A == recordSet.type()) {
      // Hosted Zone alias for the tenant subdomain
      String recordSetName = recordSet.name();
      String existingSubdomain = recordSetName.substring(0,
          recordSetName.indexOf(domainName) - 1);
      if (subdomain.equalsIgnoreCase(existingSubdomain)) {
        duplicateSubdomain = true;
        break;
      }
    }
  }
  if (duplicateSubdomain) {
    LOGGER.error("Tenant subdomain " + subdomain
        + " is already in use for this hosted zone.");
    failOnboarding(onboardingId, "Tenant subdomain " + subdomain
        + " is already in use for this hosted zone.");
    fatal.add(message);
    continue;
  }
}

We're on the weekend and a holiday here in USA, but feel free to open a Pull Request (see our contributing guide) to make this change to the OnboardingService and we can take a look when we're back.

[JaladJhamb]

Thank you @PoeppingT, finally I got to know what was the issue. Thank you once again

[PoeppingT]

@JaladJhamb for posterity and for any other users that see the same issue, can you describe briefly what your problem was and how you solved it?

[JaladJhamb]

@PoeppingT Sure I will even raise a PR for this as soon as my production related work gets over.
And just for knowledge I wanted to know, as we have installed an another SaaS in out environment and it might be using other DyanmoDB table, can we make the new SaaS to use the existing DyanmoDB as our application uses these to pull specific information needed to provision the tenant's DB, cognito user pool, S3 bucket, etc.

[brtrvn]

@JaladJhamb We'd need a little more detail on what you're doing and how you have things setup to be able to help. When you say you have "installed another SaaS" do you mean you have multiple SaaS Boost environments installed in a single AWS Account? Do you mean you have the same environment label installed in different AWS Regions in that same AWS Account? Do you mean you have some other product/software that you've setup in the AWS Account?

When you refer to DynamoDB, are you referring to the SaaS Boost Tenant Service database? Have you created a different one? How was it created? Have you modified the Tenant Service codebase? When you say "provision the tenant's DB, cognito user pool, S3 bucket" it makes me think you're not using SaaS Boost for any of the application config/tenant onboarding/workload deployment features. That's absolutely fine, but we'd need to understand what you're trying to accomplish to best be able to help you.

[JaladJhamb]

@brtrvn @PoeppingT yes we have multiple saas environment in a single account l, one is the old version i.e V0 and another one is the latest one i.e V2.1, as it was creating issue to update so we thought to install a new latest SAAS rather than updating, and as our developers are working on the lambda part, so they have actually pointed lambda to the bucket which was created when v0 was installed, and now developers need to make changes in there code to point lambda to the new DynamoDB created with v2.1, so it will really difficult for them to sort this out, so is there anyway that we can do with latest SaaS Installed that it takes things from the existing bucket rather than taking things from the bucket that it has created.
We just don't want to make any changes to our code.

[JaladJhamb]

@PoeppingT @brtrvn Any solution to this?

[JaladJhamb]

Hello @PoeppingT @brtrvn @hyandell @hariohmprasath the DynamoDB table created with the earliest versipn saas has different schema while the DynamoDB table created using the latest SaaS has different schema and I can't even append the data of the new DynamoDB table to the older one just because our lambda points on to the old DynamoDB table, please help with this if you have any solution to this.
Thanks

[brtrvn]

Hello @JaladJhamb version 2 of SaaS Boost was released, it is not backward compatible with version 1.

You will have to manually "migrate" your tenants from one install to another. First, install a clean version of SaaS Boost v2.x, then configure the application to support your workload. Push your docker image up to the new repo created by SaaS Boost and then you can onboard your tenants into the new install. You can give them the same name as you did in the prior version.

It is up to you how to migrate any other data or metadata about your tenants. Has your v1 install of SaaS Boost been running in production? What other data are you concerned about migrating?

[JaladJhamb]

@brtrvn yes our v1 install of SaaS boost is already running in production and we actually wanted another tenant into it, but while using the older version now and trying to onboard tenants again ,were getting failed so we installed a latest SaaS and the new SaaS has created table with different schema as compare to the older table.

[JaladJhamb]

So should I try to onboard the tennants again in the older SaaS, which was deployed earlier and if it get fail where i can see the reason behind it of getting failed, under which metric in cloudwatch it will show me the issue because of which the tenants are unable to onboard

[brtrvn]

If tenant onboarding is failing in your existing environment, we should figure out why it's failing. You should look through the CloudWatch Logs for the Onboarding service and/or look at the CloudFormation console.

[JaladJhamb]

Okaay @brtrvn thanks for this, i will try to onboard tenants with the older version and if I am unable to sort it, I will revert back to you
And thank you so much for helping me out

[brtrvn]

We would be happy to help you think through the manual steps to take to be able to migrate your production environment to the v2 line of SaaS Boost as it has a huge number of improvements over v1. Let us know what debugging turns up on your current v1 installation -- in particular make sure you have enough quota to onboard a new tenant (VPCs, ECS vCPU, etc...).

[JaladJhamb]

Hello @brtrvn this is the error I am getting while onboarding tenants using the older SaaS version

And We have our application in linux then why it is taking image id of windows server

[JaladJhamb]

And @brtrvn as I see in the DyanmoDB tables of the old SaaS and the new SaaS there are multiple of values which are different and some values are even being removed and in our code we have actually mentioned many things which are now removed, like overridedefaults is there in old DynamoDB table which is being included even in the code and it is not there in the new table and most of the values are being nested into the new table which is creating issue for us,
I just request you to help me in this.
Just like for the load balancer it was ALB and now it is Loadbalanacer and even it has provided nested values under it like arn, consoleURL and name,
> If I add attributes or remove it from the new table will it effect the environment for the SAAS Boost.

I was actually thinking to make my new table same as the older one

[JaladJhamb]

@PoeppingT it has worked and the tenants got onboarded successfully and I am really thankful for the help you provided.
And @PoeppingT now are task is to migrate our tenants from v0 to the latest version, and we'll be trying on our QA account for testing first as we have tenants running there.

[JaladJhamb]

@PoeppingT is there any documentation available which provides a process to migrate the tenants to the latest version of SaaS boost

[JaladJhamb]

@PoeppingT @brtrvn @hariohmprasath please do let me know if there is sny doc available to migrate old tenants to the latest version of SaaS

[JaladJhamb]

@PoeppingT please do help me in this, this has created some changes in the attributes, after successfully onboarding the tenants with the old SaaS the entry in the DynamoDB table has 3 attributes which are not mentioned when compare to the earlier ones.

[JaladJhamb]

@PoeppingT This needs to be sorted as soon as possible.Please

[brtrvn]

JaladJhamb - note that you do not have to tag Hari in your posts. He has not contributed to SaaS Boost in over a year.

To answer your question, no, there is no documentation on how to migrate tenants from v1.x of SaaS Boost to v2.x. Every SaaS application is different and what it means to migrate a tenant from one set of resources to another is defined by that application. This is out of the scope of SaaS Boost. How would you move a tenant between installations of your application today? Does it have a database? Does it have a file system? Do you need to keep the audit logs? There are so many questions and variables that influence what it means for your application to migrate tenants, SaaS Boost cannot do that for you.

From the SaaS Boost standpoint, I would probably follow a process that might include the following steps:

1. Create a fresh new install of SaaS Boost (probably in a different AWS account, but could use the same account with a different SaaS Boost environment label).
2. For each unique combo of "override defaults" you did in v1 of SaaS Boost during tenant onboarding, create a Tier in v2. For example, if your "default" in v1 was Medium Size Compute, and a min/max scaling of 2/4 and then you overrode those defaults for larger customers with XL compute and 4/10 scaling, then I'd make 2 tiers. One to represent the smaller resource consumption and another to represent the larger.
3. Configure your application in v2. By definition, this would be a single public service with a path of /* (because v1 only supported a single container). Configure the supporting resources (databases, file systems, etc.) and set the configurations for each tier you defined. You of course have more flexibility in v2 than you did in v1 so you can change things if you want.
4. For testing, I would initially leave the domain name blank in the application config so you don't get any DNS clashes.
5. Build and push your application's docker image to the new v2 generated ECR repo for the service you configured.
6. Now submit an onboarding request for each tenant being sure to select the correct tier that matches the "override defaults" setup you used in v1 and leave the subdomain blank.

At this point your application should be configured for SaaS Boost v2 and you should have successful deployments for each of your onboarded tenants. Now you need to start the manual process of "migrating" tenant data from the other environment -- whatever that means to your application. Maybe you need to dump and restore databases? Maybe you need to migrate files from one file system to another? Maybe you need to export old logs to a flat file or into S3 so you can query them? Maybe there are other things you need to do?

Once you've moved all your application requirements over, you can test the tenant environments using the AWS generated DNS entries for the load balancers and make sure everything is working properly.

After all testing is complete, you can configure DNS in the new v2 installer by setting you application's domain name and update the subdomain for the tenants and "flip the switch" from the old setup to the new.

[JaladJhamb]

Hello @PoeppingT @brtrvn, in one of our environments using the SaaS v0, I get this error, I have even check the ssm paramter and the parameter in the CloudFormation, in both there is a bucket specified.

[JaladJhamb]

@PoeppingT @brtrvn Do let me know regarding this please.

[PoeppingT]

@JaladJhamb have you made any changes to this SaaS Boost installation? What parameters do you have in SSM ParameterStore corresponding to this environment? Are you trying to upgrade this environment to v2, or just onboard a new tenant for the existing environment?

[JaladJhamb]

@PoeppingT its being sorted there was some issue with the stack which got sorted.
Thanks

[tomg-formpak]

@brtrvn @PoeppingT

We are currently on v2.1.0 and we have two tenants enabled as part of our testing. Now how to upgrade to v2.2.3 without affecting any existing settings and configurations?

• Can I point to the latest head of the release to launch the installer, running $ git branch, aws configure and $ sh ./install.sh?
• Which option to start with "Update Web application for existing AWS SaaS Boost deployment" or "Update existing AWS SaaS Boost deployment"? What is the difference between these two?
• Is there anything that needs to be checked or backed up before updating to the latest version?

Please let me know, Thank you :)